Deep Models for Multi-View 3D Object Recognition: A Review
Mona Alzahrani Muhammad Usman Salma Kammoun Saeed Anwar Tarek Helmy


Deep Models for Multi-View 3D Object Recognition: A Review]Deep Models for Multi-View 3D Object Recognition: A Review













1,2]Mona Alzahranig201908310@kfupm.edu.sa

[1,3,5]Muhammad Usmanmuhammad.usman@kfupm.edu.sa

4]Salma Kammounsmohamad1@kau.edu.sa

1,3]Saeed Anwarsaeed.anwar@kfupm.edu.sa

1,5]Tarek Helmyhelmy@kfupm.edu.sa


*[1]Department of Information & Computer Science, King Fahd University of Petroleum & Minerals (KFUPM), Dhahran, Saudi Arabia

[2]College of Computer and Information Sciences, Jouf University, Sakaka, Saudi Arabia

[3]SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd University of Petroleum & Minerals (KFUPM), Dhahran, Saudi Arabia

[4]Faculty of Computing and Information Technology, King Abdulaziz University (KAU), Jeddah, Saudi Arabia



[5]Center for Intelligent Secure Systems, King Fahd University of Petroleum & Minerals (KFUPM), Dhahran, Saudi Arabia





Human decision-making often relies on visual information from multiple perspectives or views. In contrast, machine learning-based object recognition utilizes information from a single image of the object. However, the information conveyed by a single image may not be sufficient for accurate decision-making, particularly in complex recognition problems. The utilization of multi-view 3D representations for object recognition has thus far demonstrated the most promising results for achieving state-of-the-art performance. This review paper comprehensively covers recent progress in multi-view 3D object recognition methods for 3D classification and retrieval tasks. Specifically, we focus on deep learning-based and transformer-based techniques, as they are widely utilized and have achieved state-of-the-art performance. We provide detailed information about existing deep learning-based and transformer-based multi-view 3D object recognition models, including the most commonly used 3D datasets, camera configurations and number of views, view selection strategies, pre-trained CNN architectures, fusion strategies, and recognition performance on 3D classification and 3D retrieval tasks. Additionally, we examine various computer vision applications that use multi-view classification. Finally, we highlight key findings and future directions for developing multi-view 3D object recognition methods to provide readers with a comprehensive understanding of the field.



















[
    [
    April 29, 2024
==================





§ INTRODUCTION

Deep Learning (DL) has lately been popular to solve many research problems involving image, sound, text, or graph processing. Significant advances in the DL networks have been successfully proposed and implemented for one-dimensional (1D) and two-dimensional (2D) data <cit.>. This led to remarkable results in different computer vision tasks such as recognition (classification and/and retrieval), detection and localization, segmentation, and scene understanding <cit.>. Due to the huge success of Deep Neural Networks (DNNs) in many 2D computer vision tasks, there have been recent efforts to use them for tackling tasks in more challenging scenarios and complicated settings such as Three-dimensional (3D) data processing <cit.>.

Three-dimensional data, such as 3D scenes or objects, is considered a priceless resource in the computer vision field. The reason behind that is because the 3D data, unlike the widely available 2D data, has rich information about the full geometry of the observed scenes and objects, thus providing a chance for the machines to understand the environment better <cit.>. Because different sensors capture 3D data, it comes in different representations, as shown in Fig. <ref>, e.g., polygon meshes, voxels, point clouds, multi-view renderings images, etc. <cit.>. These 3D representations vary in both properties and structure. Hence, understanding their structure is essential for analyzing the 3D objects for various applications <cit.>.



The availability of 3D data and advancements in DL has led to increased exploration by researchers in the application of DNNs for solving various computer vision problems involving 3D scene understanding, such as 3D object recognition, 3D object classification, 3D shape retrieval, detection, and segmentation <cit.>. However, extending DNNs to work with 3D data is challenging and non-trivial since DL networks have traditionally been designed to process 1D or 2D data <cit.>, and 3D objects possess complex geometric features with significant variations in their various 3D representations. As a result, DL on 3D data has emerged as a separate research field, referred to as 3D Deep Learning (3D-DL), within the broader domain of computer vision <cit.>.





        


One of the crucial domains of study within the field of 3D computer vision is the recognition of 3D objects, also called 3D shape recognition. This area of research encompasses both 3D classification and 3D retrieval techniques, as highlighted in recent literature <cit.>. The 3D object classification task involves classifying a given 3D object by identifying its category (i.e., class label) as shown in Fig. <ref>.a. In contrast, the 3D object retrieval task involves giving a 3D object as a query to find all the 3D objects in the dataset as the retrieved shapes that have the same predicted category as shown in Fig. <ref>.b <cit.>. The recent attention to building recognition models directly from 3D data has appeared for several reasons. One of them is the improvements in 3D imaging sensors (e.g., Microsoft's Kinect and LiDAR sensors) <cit.>, which made the technology practical for real-world applications such as medical image analysis, automated driving, intelligence robots, virtual/augmented reality, archaeology, and others <cit.>. Another reason is the availability of the academic datasets of 3D objects such as ModelNet <cit.> and ShapeNet <cit.>, and online repositories such as TurboSquid [ TurboSquid. Retrieved on September 17, 2023 from: https://www.turbosquid.com/Search/3D-Models.], and 3D Warehouse [ 3D Warehouse. Retrieved on September 17, 2023 from: https://3dwarehouse.sketchup.com.], which allow for training more complex recognition models than before <cit.>.


The researchers have classified previous methods of 3D object recognition into three distinct categories based on their representation of input data <cit.>: i) point-based methods <cit.>, ii) volumetric-based methods <cit.>, and iii) view-based methods <cit.>. The researchers evaluated the techniques utilized for 3D object recognition within the third classification. The justification for this focus is that view-based methods have demonstrated superior performance in 3D object recognition, thereby achieving the current state-of-the-art results <cit.>, and they are more similar to the human visual system <cit.>. In these methods, the 3D objects are rendered through their projected 2D multi-view images (i.e., views) from different viewpoints and angles, as shown in Fig. <ref>.



 §.§ Previous Surveys and Our Contributions

Many surveys are found in the literature <cit.> covering the advances of DL methods on 3D object recognition. Table <ref> summarized these surveys in terms of their published years, titles, the computer vision tasks, the targeted 3D data representations, the DL methods categorization employed for 3D object recognition, and the time period. These surveys were either focused explicitly on a singular 3D data representation, such as 3D multi-view data <cit.> or point cloud data <cit.> or even encompassed a broader scope and included a variety of different 3D data representations <cit.>. 

[p]




    0.8Summary of the existed 3D object recognition surveys

max width=



No.     Reference & Year      Year     Title     Computer Vision Tasks      3D Data     DL Method Categorization      Time Range 
 
Gray1                <cit.>                                                     2017              Deep learning advances in computer vision with 3d data: A
  survey                                  
      3D scene
  segmentation, 3D object classification and retrieval                                                                          General 3D data      Direct 3D data, descriptors, RGB-D, 2D projectionsviews,
  and hyperspectral DL methods                                                                                             Until 2016           
 

2                <cit.>     2017              3D computer vision based on machine learning with deep
  neural networks: A review                                            
      Classification and segmentation, and generation                                                                                          General 3D data       -                                                                                                                                                                                    Until 2017           
 Gray
3                <cit.>                                                        2018              A survey on deep learning advances on different 3d data
  representations                           
   
      3D object classification, segmentation, and retrieval, 3D
  correspondence                                                               General 3D data        Euclidean and non-Euclidean DL methods     Until 2018           
 

4                <cit.>                                                          2020              Deep learning for 3d point clouds: A survey                                                         
         
    3D object detection and tracking, 3D shape classification, and 3D point cloud segmentation     Point clouds          Point-based and view-based DL methods                                                                                                                                                Until 2020           
 Gray
5                <cit.>                                                  2020              A review on deep learning approaches for 3d data
  representations in retrieval and classifications 
  
        
      3D object classification and retrieval                                                                                                   General 3D data       Raw data, surfaces,
  solids, high-level 
  structures, and multi-view DL methods        Until 2020           
 

6                <cit.>                                                            2021              Review of multi-view 3d object recognition methods based
  on deep learning                        
                                                      
      3D object classification and retrieval                                                                                                   Multi-view             Model-based and view-based CNN
  methods                                                                                                               Until 2021           
 Gray
7                Our Survey                                                                                              2023              Deep Models for Multi-View 3D Object Recognition: A Review                    
  
      3D object classification and retrieval                                                                                                   Multi-view                              Model-based and view-based CNN
  methods      Until present        












[p]


Competitive analysis of the existed 3D object recognition surveys and their covered topics

max width=

 

No.     Covered Topics      <cit.>     <cit.>     <cit.>     <cit.>     <cit.>     <cit.>     Our Survey 
 
  Gray1       3D Data Categorization                                                                                                           
 
2       3D Object Recognition
  MethodCategorization                                                                                        
 Gray
3       General 3D Object Recognition Pipeline                                                                                                          

4       Summary of 3D
  Datasets                                                                                                       
 Gray
5       Comparison of
  the Methods Based on Experimented 3D Datasets                                                                                                                                    
 
6       Summary of Camera
  Configurations                                                                                                                                                     
 Gray
7       Comparison of
  the Methods Based on Experimented Camera Configurations                                                                                                                          

8       Summary of Views Selection Strategies                                                                                                                                               
 
Gray
9       Summary of Pre-trained
  CNN Architectures                                                                                                                                              
 
10      Comparison of
  the Methods Based on Experimented Pre-trained CNN Architectures                                                                                                                 
 
  Gray
11      Summary of Fusion
  Strategies                                                                                                                                                                   
 
12      Summary of Evaluation
  Metrics                                                                                                                                                        
 
  Gray
  13      Comparison of the Methods Based on Recognition Performance                                   
 
14      Relevant
  Computer Vision Applications                                                                                                                                                          
 Gray
15      Factors impacting the recognition performance                                                                                                                                                     
   
16      Limitations and future research                                                                                                                       
   













Regarding the surveys that generally covered different 3D data representations <cit.>, <cit.> review the existing DL methods applied to 3D data. Specifically, the authors surveyed the main techniques that tackle the 3D data manipulation steps, which include segmenting the 3D scene, detecting the 3D key point, extracting the 3D descriptors, and retrieving and classifying the 3D object. They also detailed the most utilized architectures in 3D computer vision tasks, which are Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Deep Belief Networks (DBNs) and Autoencoders (AEs). The DL methods were classified into five categories based on the 3D representation utilized as input to the deep neural network. These 3D representations are descriptors extracted from 3D and RGB-D data by separating depth and color channels, direct access to the 3D data, one or more 2D projections/views of the 3D object, and HyperSpectral data.

For each 3D data representation, the authors overview the most popular methods for 3D shape classification and 3D shape retrieval. Similarly, <cit.> provide a review of the general DL methods in the computer vision field, highlighting Classification, segmentation, and generation tasks that involve 3D visual data. In addition, the different deep networks that have been suggested for 3D vision understanding are summarized. Also, the intersection between this field and neuroscience and psychology has been explained to understand the human visual system better. The authors expanded their review with the current major challenges in this field and concluded their survey with future research directions that were important in that year. In 2018, <cit.> categorized the 3D data representation into two main categories. The first category is the 3D Euclidean representations, which have a fundamental grid structure that accepts a common coordinates system and a global parameterization such as 3D descriptors, multi-view data, 3D projections, RGB-D data, and volumetric data (voxel and octree). The second category is the non-Euclidean representations, where no global parameterization lacks the gridded array structure, such as point clouds, graphs, and polygon meshes. The authors also detailed the DL techniques for different 3D data representations under these two categories. In 2020, <cit.> reviewed the performance of DL methods on several 3D data representations categorized as solids (e.g., octree and voxels), surfaces as a polygon mesh, multi-views, high-level structures (e.g., 3D descriptors and graph), and raw data (e.g., RGB-D data, point cloud, and 3D projections). The performance of these representations was reviewed for 3D object retrieval and classification tasks. The same year, in 2020, <cit.> presented a review covering three tasks, including 3D object detection and tracking, 3D shape classification, and 3D point cloud segmentation. It mainly focuses on the recent progress of DL methods, which take point cloud as the 3D data representation, and only a few multi-view-based techniques were mentioned in that work. The point-based methods were categorized based on the network architecture utilized for the learning, including pointwise MLP, graph-based, hierarchical data structure-based, convolution-based, and other standard methods. 



Regarding the surveys that focused on the 3D multi-view data, <cit.> presented an inclusive review of studies published until 2021 on the latest progress in the DL methods employed for multi-view 3D object recognition. The authors divided these methods into three categories depending on their input: voxel-based methods, pointset-based methods, and view-based methods, where they focus on the last category. They introduce the most used 3D datasets and the evaluation metrics for classification and retrieval. Also, they provide an analysis of the existing methods in terms of the utlized viewpoints setting and input modes, backbone network, feature extraction, fusion strategies, and viewpoints selection mechanisms. Towards the end, they summarized the performance of the existing methods on several famous datasets.




Prior studies, as seen in Table <ref>, have either encompassed a broad and outdated range of 3D data representations <cit.> or solely focused on a single aspect of 3D data representation, excluding 3D multi-view data <cit.>. The survey that focused on the 3D multi-view data <cit.> also covering different DL methods and according to what they mention in their paper was conducted on a very narrow time range from 2018 to 2021 as shown in Fig. <ref>. In addition, based on the competitive analysis of the existing 3D object recognition surveys presented in Table <ref>, different surveys lack various importance topics such as 3D data categorization <cit.>, 3D datasets summarizing and recognition performance comparison <cit.>, multi-view camera configurations, Views selection strategies, pre-trained CNN architectures, evaluation metrics <cit.>, fusion strategies <cit.>, factors impacting the recognition performance <cit.>, future works <cit.>, relevant computer vision applications <cit.>, and the most important the general pipeline of the multi-view based methods and its steps <cit.>. Therefore, in-depth evaluation of the most efficient 3D object recognition techniques from their inception is imperative, specifically focusing on 3D multi-view data representation and utilizing the popular and widely applied DL networks.



In comparison to existing surveys <cit.>, the primary contributions of this research can be summarized as follows:


    
  * This review paper comprehensively covers the most recent progress of DL-based and transformer-based 3D object recognition methods on 3D multi-view data, highlighting the state-of-the-art techniques in the field and summarizing them for the readers. 
    
  
  * Unlike existing reviews <cit.>, this research specifically focuses on DL-based and transformer-based methods. Among these methods, it focuses on the techniques that take 3D multi-view data as input representation.
    
  
  * It presents the analyses of the general pipeline for DL-based multi-view 3D object recognition, including a summary of the various techniques employed at each stage of the pipeline.

 
  * It contains the last developments of the transformers and attention mechanisms in the multi-view 3D object recognition since 2021, which are not covered by the existing reviews <cit.>.
    
    
  * The comprehensive details of existing DL-based and transformer-based multi-view 3D object recognition models are analyzed and discussed in terms of the primarily used 3D datasets, camera configurations and the number of views, view selection strategies, pre-trained CNN architectures, fusion strategies, and recognition performance (e.g., Tables <ref>,  <ref>,  <ref>, <ref>, and <ref>).
    
   
  * In addition, this review showcases various computer vision applications that incorporate multi-view classification through the utilization of CNNs. Furthermore, contrary to the survey <cit.> that has a narrow time range, this review provides an overview of the advancements in multi-view 3D object recognition methods, starting from the inception in 2015 until the present day as seen in Fig. <ref>.

    
   
  * To provide the readers with future directions and ideas, and several facts about how the various factors impact the recognition performance and future trends for the development of multi-view 3D object recognition methods are highlighted in terms of viewpoints and number of views, backbone and feature extracting, feature fusion and view selection, light direction and object color, and number of transformer blocks.
     




 §.§ Paper Organization

The remainder of this paper is organized as follows. In Section <ref>, different 3D data representations are introduced, multi-view learning is defined, more details on the 3D multi-view data and the use of multi-view in computer vision fields are presented, and the reasons that led to most of the 3D object recognition methods to choose the multi-view among all the deep architectures are specified. Sections <ref> and <ref>, the cores of this paper, start by comparing different 3D object recognition methods, emphasizing the analysis of up-to-date DL-based and transformer-based multi-view recognition methods. Section <ref> covered various relevant computer vision applications that employed the DL-based multi-view for image classification. Section <ref> provides the readers with ideas where several recommendations and directions for developing multi-view 3D object recognition methods are highlighted. Finally, this review is concluded in Section <ref>.




§ OVERVIEW


This research reviews the 3D object recognition methods with multi-view 3D data representation. Hence, in this section, some background is provided to the readers to help them understand the different 3D data representations in general and multi-view data representation in specific as it is the concern representation in this study. In addition, it gives an overview of multi-view learning from computer vision and machine learning perspectives. In comparison, the latter focuses on the multi-view conventional neural network.



 §.§ 3D Data Representation


Due to the high significance of 3D-DL in complex tasks such as self-driving cars, robotics, and virtual and augmented reality, it has attracted increasing attention in the last few years <cit.>. However, in the 3D-DL field, two fundamental challenges exist: i) the 3D data representation and ii) the adopted network architectures. To get a successful deep model, the usability, simplicity, and efficiency of the 3D data representations and their fundamental properties need to be considered <cit.>. Hence, various 3D data representations, as shown in Fig. <ref>, were used by different 3D-DL methods.

3D data come in different representations depending on the scanning device used to capture the 3D object. Therefore, the 3D representations vary in structure and properties <cit.>. Several 3D representations encouraged the researchers to exploit these data on their tasks and select the suitable one <cit.>. These representations can be categorized into two main categories based on <cit.> taxonomy. The first category is the 3D Euclidean representations, which have a fundamental grid structure that accepts a common coordinates system and a global parameterization such as 3D descriptors, multi-view data, 3D projections, RGB-D data, and volumetric data (voxel and octree).

On the other hand, the second category is the non-Euclidean representations, where there is no global parameterization that lacks the gridded array structure, such as point clouds, graphs, and polygon meshes. Another 3D data categorization is the taxonomy introduced by <cit.>. They classified the 3D data into five categories as follows: i) Raw data includes the 3D projections, RGB-D data, and point cloud; ii) Solids representations such as voxels and octree; iii) Surfaces that represent the polygon meshes as the most popular representation in this category, iv) Multi-views data, and v) High-level structures such as 3D descriptors and graph. Despite these different taxonomies, each 3D data representation has a typical structure, properties, ways for acquisitions, advantages, and limitations, as summarized clearly by <cit.>. It should be noted that all of these 3D data representations are vastly grown research areas and no direct winner among them <cit.>. But each has its advantages over the other, and choosing the suitable one is crucial to the performance of a particular method.



 §.§ Multi-View Learning


Multi-view Learning (MVL) is a technique that involves the utilization of classifiers to extract common patterns or feature spaces from various separate feature subsets or sources of the same data <cit.> and is based on the acquisition of data from multiple sources, which results in the creation of various views. For example, multi-media data can be represented as video and audio and website data as text, images, and hyperlinks. Moreover, due to the advancement of data technology in real-world applications, vast amounts of multi-view data are generated. Compared to the single-view learning methods, data from various views mostly contain complementary information; hence, MVL utilizes this to learn more comprehensive representations <cit.>.

MVL has obtained significant importance in the last decades due to its remarkable performance in machine learning and computer vision fields <cit.>. In machine learning, it has been successfully applied in several areas, such as clustering, supervised learning, ensemble learning, semi-supervised learning, dimensionality reduction, reinforcement learning, etc. In computer vision, MVL has achieved significant performance in applications such as image recognition, handwritten digit recognition, human gait recognition, etc. <cit.>. The rest of this section presents multi-view use in computer vision fields and more details on the 3D multi-view data. In addition, the reasons that led most of the 3D object recognition methods to choose the multi-view CNN among all the deep architectures are specified.



  §.§.§ Multi-view Computer Vision


In the computer vision field, the view refers to an image displaying the same object sample (as shown in Fig. <ref>) or part of it (as shown in Fig. <ref>) from a specific perspective. The collection of images is the combination of different images (or views) that were captured from the same object but different viewpoints (from different cameras) <cit.>. 



A significant task of the multi-view computer vision field is the multi-view classification, which is motivated by human behavior. For example, a typical use-case for multi-view classification is plant species identification (see Fig <ref>), where a human watches different features of an unfamiliar plant to be able to identify its species. Because it could be hard to distinguish between different plant species by their flowers alone, it requires additional views, e.g., their leaves <cit.>.

<cit.> categorized the computer vision systems that have multiple camera views in different ways. One category was based on the number of views as shown in Fig. <ref> .a and summarized as follows: i) Single view system (Fig. <ref> .a.1) when it uses only one camera, ii) Stereo view system (Fig. <ref> .a.2) when it uses two cameras that are necessarily next to each other, and iii) Multi-view system (Fig. <ref> .a.3) when the system uses more than two cameras which is our concern in this review. Another important category was based on Fields of View (FOV) overlapping as shown in Fig. <ref> .b and listed as follows: i) Overlapping fields (Fig. <ref> .b.1) when the cameras are focused on the same targeted object, ii) Non-overlapping fields (Fig. <ref> .b.2) when the cameras not capturing the same object on two simultaneous views, and iii) Small overlapping fields (Fig. <ref> .b.3) when the views are complete each other.





  §.§.§ Multi-view Convolutional Neural Network

 
Due to their powerful feature extraction ability, DL methods have succeeded in many application domains with excellent performance, such as artificial intelligence and computer vision. DL methods can efficiently learn the targeted data's non-linear, abstract, and complex representations by utilizing several hierarchical layers <cit.>. Also, the deep MVL methods have been progressively used with encouraging results <cit.>. And they enhanced the performance of the traditional machine learning algorithm <cit.>. <cit.> provides a comprehensive review of deep MVL methods; however, the CNN-based multi-view models are the most extensively utilized DL-based models in computer vision. The standard CNNs can be employed for scene labeling, face recognition, image classification, human pose estimation, action recognition, document analysis, etc. <cit.>. CNN represents an advanced method and works equally and sometimes better than humans in various tasks, especially for classification problems <cit.>. In addition, they significantly outperformed the traditional machine learning methods in a broad range of tasks, especially for image analysis <cit.>. CNN is a neural network that contains several layers. Its layers involve two roles: the first is feature extraction (feature learning), and the second handles the classification <cit.>. CNN consists of a mixture of three main types of layers (see Fig.<ref>):



  * Convolutional layers: It is the first step of feature extraction. In these layers, several utilized filters (a.k.a. kernels) are used to convolve the whole inputted image or the output of the previous layer to extract and distinct several features and generate the different feature maps <cit.>. Filters are also called feature detectors because they can detect the features within an image, such as vertical and horizontal lines <cit.>. So, the feature maps are generated as the output of the convolutional layer.
    

  * Pooling or subsampling layers: It is responsible for reducing the size or spatial dimensions (width × height) of the inputted feature maps, which is why it is called downsampling or subsampling. This size reduction is beneficial for reducing the network's computational costs and avoiding overfitting <cit.>. This layer does not impact the depth dimension of the feature maps, resulting in the same feature map number passed to the next layer as input <cit.>. The pooling operation is precise in the features that exist in the region of the feature map <cit.>. There are different types of pooling operations, but the most frequently utilized are max pooling or average pooling <cit.>. So, the pooled feature maps are generated as the output of the pooling layer.  


  * Fully Connected (FC) layers: After a stack of multiple convolutional layers and pooling layers, one or more FC layers <cit.>. In the FC layer, neurons have full connections to all activation functions in the previous layer <cit.>. The purpose of FC layers is to transfer the 2D pooled feature map into a flattened structure to get a single 1D feature vector for further processing <cit.>. Usually, in classification and recognition tasks, the final FC layer is connected to a classifier such as softmax that produces the response, e.g., class label, to the initial inputted image <cit.>.
    








As a typical single-view CNN architecture, it aims for feature representation learning with multiple parameter optimization, which in various domains has exhibited excellent performance <cit.>. On the other hand, the multi-view CNNs have access to multi-view information of the same data, which aims to integrate helpful information from multiple views to learn more comprehensive and discriminative representations to yield a more effective classifier <cit.>. The current multi-view CNN architectures are classified into the following two types as illustrated in Fig. <ref>: i) One-view-one-network strategy (Fig. <ref>.a) that uses one CNN for each view to extract feature representation separately, then the multiple representations from all the views are fused (merged) through the later part of the network <cit.>. Several efforts have been made to design multi-view CNN architectures with this type of strategy (e.g., the famous MVCNN <cit.>, RotationNet <cit.>, and GVCNN <cit.> models) for 3D object recognition; and ii) Multi-view-one network strategy (Fig. <ref>. b) that uses a single CNN to feed it with multi-view data to get the final representation <cit.>. An example of this approach is the architecture of <cit.>, which was proposed for breast cancer classification.





The CNN-based methods have been extensively utilized for multi-view 3D object recognition for several reasons. One reason is that multi-view 3D object recognition is based on image classification, which is a hard problem that requires better image classification techniques. The CNN shows robust performance in automatic feature extraction and classification, which makes it a broadly employed classifier <cit.>. Also, CNNs are invariant to transformations, which means they can detect the class label even if we translate the inputs. This property is useful for specific computer vision applications, including the classification tasks <cit.>. The CNN-based methods proved to have improved feature representative capability than the standard methods for recognition problems <cit.>. Despite this, CNNs still suffer from some limitations. It heavily depends on the availability of labeled data. In addition, insufficient or small training data, even after data augmentation, can result in poor performance and overfitting problems <cit.>. Furthermore, in the training phase, it is computationally expensive <cit.> and needs to learn a large number of parameters <cit.>. To avoid this limitation, previous multi-view 3D object recognition approaches have been using transfer learning methods, data augmentation, and pre-training CNN architecture such as AlexNet <cit.>, VGG-M <cit.>, VGG-VD <cit.>, GoogLeNet <cit.>, or ResNet <cit.> that already initialized with pre-trained parameters instead of setting ones <cit.>.

Among the CNN-based 3D object recognition methods, the view-based networks for 3D object recognition have performed best to achieve the current state-of-the-art performance. Using the DL technique for multi-view 3D object recognition has become one of the rigorously researched directions due to the advanced and successful pre-trained classification network that can be used directly as the backbone network, and multiple viewpoints render multiple views of the object, which can complete each other’s detailed. However, there still exist some challenges in this area <cit.>. Hence, many DL-based methods have been designed to solve the problems relating to this research direction. 


The first idea is to collect information from multiple 2D views of a 3D object into a global shape descriptor for 3D object recognition via a novel CNN architecture called Multi-View Convolutional Neural Network (MVCNN) <cit.>. In that work, <cit.> proved that 3D objects can be represented effectively using view-based 2D representations, which could be utilized for classification and retrieval. They contribute to the field by generating global shape descriptors and leveraging image and 3D shape datasets to improve performance. 



§ DL-BASED MULTI-VIEW 3D OBJECT RECOGNITION



This review focuses on the recognition of 3D objects, which encompasses both 3D object classification and retrieval tasks because these tasks were the first to be effectively addressed using the MVCNN method <cit.>. MVCNN method later served as the milestone for using deep 2D CNNs for multi-view classification <cit.>. In addition, with the development of 3D data and their processing techniques in computer vision, 3D object recognition become an essential task with a critical role in wide applications <cit.>.

Methods of 3D object recognition using DL have been classified into three categories based on the input data representation (see Fig <ref>) <cit.>:



  * Pointset or point-based methods: in this category, strategies are implemented directly on the 3D objects, which are usually represented as point clouds or sets of unordered points <cit.>. The point cloud is a collection of points in the 3D space used to describe the geometric shape of the object’s surface. Each point is defined by 3D coordinates (x, y, and z), gray value, RGB color value, and depth data <cit.>, which was later analyzed by a 3D-DL network to learn the features for recognition <cit.>. To generate point cloud representation, devices such as LiDAR and 3D scanners can be utilized to generate laser light that hits the object’s surface. It is worth noticing that the 3D reconstruction is the process of reconstructing the number of points of the 3D object as shown in Fig. <ref>.b, and more point clouds led to more created accurate models <cit.>. Some of the methods achieving improved performance are PointNet <cit.> as the first point cloud CNN model, PointNet++ <cit.>, PointCNN <cit.>, RS-CNN<cit.>, and DGCNN <cit.>.


  * Voxel or volumetric-based methods: point cloud data can be used to construct a 3D regular grid representation called a voxel. In 3D space segmentation, the voxel represents the tiniest unit of digital data. Each unit can be seen as a fixed coordinates grid. It also has a resolution as the 2D image; the divided 3D space with better quality led to a smaller grid with better resolution <cit.> as seen in Fig. <ref>.c. For the first category, these methods were implemented directly on the 3D objects, which usually represented them as a collection of voxels in the 3D Euclidean space <cit.>, and then analyzed them by a 3D-DL network to learn the features for recognition <cit.>. Most popular voxel-based methods are: 3D ShapeNets <cit.> as the first volumetric CNN model, VoxNet <cit.>, DLAN <cit.>, LightNet <cit.> and VRN <cit.>.


  * View-based methods: Multi-view images are the oldest representation of the 3D data <cit.>. These methods rendered the 3D objects through its projected 2D multi-view images (also called views) from different viewpoints and angles as shown in Fig. <ref>.a. In this method, the 3D geometry is expressed more simply in <cit.>. Then, feed these views, which are the input data to the 3D-DL network (mostly CNN) to gather multi-view features later fused as global descriptors utilized for recognition <cit.>. Among these networks, the milestone view-based methods for 3D shape recognition are MVCNN <cit.> as the first multi-view CNN model, GVCNN <cit.>, RotationNet <cit.>, View-GCN <cit.>, and MVTN <cit.>.



       


Both first and second categories fall under one umbrella called model-based methods <cit.> which convolve a 3D shape in the 3D space using a 3D convolution filter, consequently producing a 3D representation straightly from the 3D data. Although these model-based methods are effective and use the spatial information and all of the information about the object when the 3D representation on a full resolution is used <cit.>, however, they suffer from some disadvantages such as: i) Time complexity: voxel resolution which needs to be significantly reduced to use for training a deep network reasonable amount of time with the available samples <cit.>, ii) Computational complexity: voxelization of the shape surface caused computational complexity and data sparsity <cit.>, iii) Memory complexity: 3D CNNs are allowed to be used in voxel-based networks, however, it suffers from cubic memory complexity <cit.>, iv) Application limitation: the computational cost limits their practical applications <cit.>, v) Point clouds are sparse, disordered, and express limited data information <cit.>, and vi) Voxel with no rotational invariance <cit.>.


Since the method’s effectiveness is dependent on the chosen 3D data representation <cit.>, this review is focused on the view-based approaches that are used for the classification and retrieval tasks for the following advantages: i) They have performed best so far among all the 3D object recognition methods <cit.>, ii) They achieved excellent accuracy for 3D shape classification because it takes advantage of the pre-trained 2D classification networks <cit.> with larger image datasets (e.g. ImageNet <cit.>), iii) They do not depend on the complicated 3D features <cit.>, iv) Capturing the inputs of these methods which are the views is easy compared to other methods which take polygon mesh or point cloud structure as input <cit.>, v) The views captured from several viewpoints can complete each other’s detailed features of the object in case of occlusion <cit.>, vi) They can be adapted to specific applications to produce feature representations from multi-view images that have superior performance compared to other features in a variety of setups <cit.>, vii) They can efficiently learn an object representation that gathers information from any inputted views number without any particular order, and continually outputs a global descriptor of the same size <cit.>, viii) A large amount of both 2D image and 3D object datasets can be used to train the view-based methods <cit.>, ix) They are more similar to the human visual system, which is fed with a flow of rendered 2D images rather than more detailed 3D representations <cit.>, x) They solve a 3D task using 2D deep architectures which narrow the gap between 3D and 2D learning <cit.>, and xi) 3D data with high resolution can be utilize due to the low memory requirement of the rendered 2D images <cit.>.


In 2D computer vision, the data structure is very clear-cut, which is images constructed of dense pixels that are ordered carefully and evenly into a precise grid. However, the 3D data world has no such consistency. 3D data can be represented as multi-view image sets, voxels, point clouds, meshes, etc. So, choosing a suitable data representation is half the battle if it will be used in the 3D-DL research field. Consequently, in the following subsections, based on what is mentioned above, this review work focused on the view-based approaches employed for 3D object recognition. Although DL-based methods have been varying in their architecture design, all the DL-based multi-view 3D object recognition models share a general pipeline described in the next subsection in detail. In addition, different 3D datasets, camera configurations, view selection approaches, pre-trained CNN architectures, and fusion strategies used with these approaches are also reviewed. 




 §.§ General Pipeline


Fig. <ref> shows the general pipeline of any DL-based multi-view 3D object recognition model. At the beginning of the model, 3D objects from 3D datasets are needed for training purposes. Then, several virtual cameras are placed around the 3D object to render the views. These virtual cameras are placed according to pre-defined camera configurations and settings. And the number of the rendered views is the same as the number of these virtual cameras. After that, each rendered view is fed to a backbone network, which is pre-defined CNN for feature extraction. Then, feature fusion must be performed using all the extracted features to create a global shape descriptor. This global descriptor is utilized later by the same pre-defined CNN for recognition. The recognition performance can be measured depending on the recognition tasks (classification or retrieval) using well-known evaluation metrics. It is worth mentioning that there is another step on the pipeline, which is view selection strategy that removes irrelevant and redundant views. The view selection step is not shown in Fig. <ref> because its place changes according to the type of selection, which is discussed later. In this paper, each step of this pipeline is detailed in subsequent subsections to describe the different choices of each step and to review the DL-based multi-view models accordingly. 









  §.§.§ 3D Object Recognition Datasets


Real-world and synthetic datasets are essential for conducting fair experiments and comparisons between various learning models <cit.>. However, supervised learning requires a large amount of labeled data. Several labeled image datasets are available, but they represent the objects by only a single image instead of a collection of views. The limitation of the unavailable collection data has prompted some studies to explore implementing new multi-view or 3D datasets as alternative sources <cit.>. These 3D datasets are ModelNet10 and ModelNet40 <cit.>, ScanObjectNN <cit.>, ShapeNet core55 <cit.>, RGB-D <cit.>, and MIRO <cit.>. Table <ref> summarizes the details of each of these datasets in terms of their types, number of classes, number of samples, and the standard splitting of samples for training, validation, and testing. Table <ref> shows the view-based related works that experimented with these datasets for 3D object recognition. Due to space constraints within the table, and  for easy access, Table <ref> in Appendix <ref> presents a concise overview of these view-based deep models, along with their corresponding references.

ModelNet is a large-scale dataset of 3D objects provided by <cit.>.  The dataset currently contains clean 127,915 3D computer-aided design (CAD) models belonging to 662 object categories. The 3D CAD models were obtained by querying each object category from online search engines such as 3D Warehouse [ 3D Warehouse. Retrieved on February 17, 2022, from: https://3dwarehouse.sketchup.com.] and manually labeling the data. Examples of 3D objects from the ModelNet dataset are shown in Fig. <ref>. The ModelNet dataset has two subsets, which are ModelNet-10 and ModelNet-40 that are mostly used in semantic segmentation, object classification, and recognition tasks <cit.> with standard splitting for training and testing. In addition to the original “unaligned” ModelNet40 and ModelNet10, both datasets have the “aligned” version that is publicly available on the ModelNet website [ Princeton ModelNet. Retrieved on May 8, 2022, from: https://modelnet.cs.princeton.edu/.]. ModelNet40 <cit.> contains 12,311 manually cleaned 3D objects (split as 9,843 for training and 2,468 for testing) that belong to 40 class categories such as chair, airplane, guitar, and cup. These 3D objects are without color information, and there is a diverse number of objects across diverse categories. Modelnet40 contains 40 categories of 3D CAD models and is a common dataset for evaluating the classification and semantic segmentation of 3D-DL models <cit.>. On the other hand, ModelNet10 <cit.> contains 4,899 manually cleaned 3D objects (split as 3,991 for training and 908 for testing) that belong to 10 class categories. The current state-of-the-art performances on 3D object classification, when the ModelNet40 benchmark dataset was experimented with, are reported by ViewFormer <cit.> with 98.90% overall accuracy among all the classification task methods, where the ViewFormer learning model is based on the multi-view approach. On the other hand, the current state-of-the-art performances on 3D object classification when the ModelNet10 benchmark dataset was experimented with, DRCNN <cit.> reported 99.34% overall accuracy, which is also a learning model on the multi-view approach.


ScanObjectNN is a point cloud dataset recently released by <cit.>. This dataset has been used for 3D classification. It is more challenging and realistic than ModelNet40 because it considers occlusions and includes background. The dataset contains 2,902 point cloud objects collected from real-world indoor scans that belong to 15 class categories. Examples of 3D objects from the ScanObjectNN dataset are shown in Fig. <ref>. Three main variants of this dataset exist to represent different levels of complexity to explore the classification methods' robustness. These variants are OBJ_ONLY (only segmented objects), OBJ_BG (objects are captured with their background), and the most challenging variant is PB (with rotating, translating, and scaling the ground truth bounding boxes). This dataset has been utilized by MVTN <cit.> to test the generalization capability of the learning model in more realistic settings.  



ShapeNet Core55 <cit.> is a subset of a large-scale 3D image ShapeNet dataset <cit.> that was developed by a team of researchers from the TTIC Institute at Princeton and Stanford universities. It comprises 51,162 clean 3D mesh objects belonging to 55 class categories, further subdivided into 203 subcategories. Different categories have different numbers of objects. The 51,162 objects are split into 35764 for training (70%), 5133 for validation (10%), and 10265 for testing (20%). ShapeNet Core55 is a challenging 3D dataset designed annually for a shape retrieval challenge called Shape Retrieval Contest (SHREC). ShapeNet Core55 was provided in the SHREC’17 track 3. Two versions of the ShapeNet Core55 dataset exist <cit.>: i) normal dataset (consistently aligned), and ii) perturbed dataset (randomly rotated). 3D model samples of the ShapeNet Core55 dataset are shown in Fig. <ref>. The ShapeNet Core55 is commonly employed for retrieval tasks as done by PANORAMA-ENN <cit.>, SeqViews2SeqLabels <cit.>, <cit.>, MVSG-DNN <cit.>, G-CNN <cit.>, and HMVCM <cit.>. However, a few works, such as 3D2SeqViews <cit.> and View-GCN <cit.>, have been using it for evaluating their models for classification tasks.

RGB-D dataset by <cit.> contains real images captured from multiple viewpoints of objects placed on a one-dimensional rotation table. This dataset includes RGB and depth multi-view images of 300 object samples belonging to 51 categories. The objects are captured from 30^∘, 45^∘, and 60^∘ elevation angles above the horizon to end up with 250,000 RGB-D images. However, the RGB-D dataset has some drawbacks that make it not suitable for training multi-view-based CNNs, which are i) the number of object samples per category is insufficient (a minimum of 2 for training), ii) it conflicts with the upright orientation assumption where the viewpoints of object samples are inconsistent against the rotation axis in some categories and iii) it does not capture the bottom faces of the objects on the turning table <cit.>. Samples of objects from the RGB-D dataset are shown in Fig. <ref>.



Multi-view images of rotated objects (MIRO) is a self-made dataset introduced by  <cit.> to test their RotationNet model. This dataset provides multi-view images captured of rotated objects using Ortery’s 3D MFP studio. It contains 120 objects belonging to 12 class categories. Each category has ten objects. At the same time, each object was captured with ten elevation angles and 16 azimuth angles to obtain a total of 160 views. However, this dataset has only been used by RotationNet <cit.>, and it doesn’t consider a mainstream dataset yet <cit.>. Samples of multi-view images of 3D objects from the MIRO dataset are shown in Fig. <ref>.








  §.§.§ Camera Configurations


The multi-view networks for 3D object recognition captured the views using some of the five mostly experimented pre-defined viewpoints shown in Fig. <ref> (also called camera settings or multi-view configurations). These five viewpoints are:

    
  * Circular configuration: the first camera setup is the regular circle. Where in this setup, as shown in Fig. <ref>.a, virtual cameras are uniformly placed on a horizontal circular path around the tested object and raised with elevation ϕ equal to 30 from the ground level and directed at the centroid of the object’s mass <cit.>. The weighted average is calculated as the object’s centroid where the weights are the face areas <cit.>. This setup is commonly beneficial to capture views of real and aligned objects initially acquired with one-dimensional turning tables. In other words, it is useful when the objects are assumed to be with an upright orientation by a consistent axis (e.g., z-axis) as the rotation axis that identified the upright orientation where the virtual cameras are distributed over 360 at intervals of the azimuth angle Θ around the axis <cit.>. In most of the related works such as <cit.>, they set the azimuth angle Θ equal to 30 as default, which means locating 12 virtual cameras that extracted 12 rendered views from an object that is inputted simultaneously. Or even set the azimuth angle Θ equal to 45 as in <cit.> that extracted eight rendered views from an object.
    
    
  * Spherical (dodecahedral) configuration: the second camera setup is the irregular spherical where this setup is without the consistent upright orientation assumption of shapes when the objects are unaligned and not in the same vertical direction <cit.>. As shown in Fig. <ref>.b, irregularly placed virtual cameras with equal spaces on a vertex of a dodecahedronsphere surrounding the object <cit.>. In this configuration, the camera viewpoints can be equally spread in 3D because a dodecahedron has the greatest vertices among regular polyhedral <cit.>. View-GCN <cit.> experimented with this configuration by locating 20 virtual cameras on the dodecahedron's vertices surrounding the object to render 20 views. Also, RotationNet <cit.> was experimented with this configuration but with three diverse patterns of rotation degrees for each of the 20 viewpoints corresponding to the three edges connected to each vertex to yield 60 candidates' viewpoints. On the other hand, MVCNN <cit.> extracted four rendered views from each one of the 20 viewpoints using the following degrees rotation 0, 90, 180, and 270 to obtain a total of 80 views <cit.>.
    
    
  * Random configuration: this setup is flexibly extended from the previous spherical configuration where 12 irregular views were randomly selected by perturbed in coordinates from the previous spherical configuration as shown in Fig. <ref>.c <cit.>.
    
    
  * Circular with elevation configuration: this setup is extended from the previous circular configuration. Where several elevation levels of viewpoints exist, as shown in Fig. <ref>.d. This configuration is also based on the upright orientation assumption as the first configuration. However, the elevation angle ϕ is not fixed and it is in [-90, 90] where it ends up with a total of 360/ Θ viewpoints <cit.>. 
    
    
  * Cylinder (panoramic) configuration: a cylinder setup is a configuration whose axis is aligned to one of the principal axes of coordinate space (e.g., the z-axis). A panoramic view is then obtained by projecting the 3D model to the lateral surface of a cylinder of radius R and height H equal to 2R. The cylinder is centered at the origin as in Fig. <ref>.e <cit.>.
    

    

Tables <ref> and <ref> summarize the experimented camera configurations and the experimented numbers of views, respectively, since they are essential factors that affect the performance of the recognition model <cit.>. From Table <ref>, it could be noticed that circular and spherical are the most common configurations that were experimented with by 25 and 14 related works, respectively, out of 37 works. Even more, from Table <ref> also, it is obvious that out of these works, 30 works have experimented with 12 views, 18 works have experimented with six views, 13 works have experimented with three views, and ten works have experimented 20 views. Decreasing the number of views, such as 1 or 2, will decrease the classification performance because much of the information will be lost, which is reasonable. However, when the number of views increases, such as 4, 6, 8, or more, the classification performance increases fast and becomes much more stable <cit.>. Even more, using 20 views, the state-of-the-art performance was achieved by RotationNet <cit.>, OVPT <cit.>, MVT <cit.>, view-GCN <cit.>, and ViewFormer <cit.>. Otherwise, worse classification performance is achieved when the model randomly selects the views. If these selected views are randomly captured from a close or identical direction, this will significantly reduce the performance because it will have the same effect as using a few views <cit.>.

[p]



3D datasets that have been experimented with the view-based 3D object recognition methods

max width=
    
    0.5


No.     3D Dataset:                       ModelNet40     ModelNet10     ScanObjectNN     ShapeNet Core55     RGB-D     MIRO
 
Gray 1      MVCNN                                                                                                                               
 

2      Pairwise                                                                                                        
 Gray

3      VolumetricMVCNN                                                                                                            
 

4      GIFT                                                                                                            
 Gray

5     PANORAMA-NN                                                                                                    
 

6      PANORAMA-ENN                                                                                                                          
 Gray

7     MHBN                                                                                                             
 

8      GVCNN                                                                                                                                  
 Gray

9      MVCNN(GoogLeNet)                                                                                                           
 

10     SeqViews2SeqLabels                                                                                         
 Gray

11     RotationNet                                                                    
 

12     Ma et al.                                                                                    
 Gray

13     VERAM                                                                                                           
 

14     MVCNN-new                                                                                                                              
 
Gray
15     iMHL                                                                                                                                   
 

16     MVSG-DNN                                                                                                        
 Gray

17     RN                                                                                                              
 

18     MLVCNN                                                                                                                                 
 
Gray
19     G-CNN                                                                                                           
 

20     DeepCCFV                                                                                                                              
 Gray

21     3D2SeqViews                                                                                           
 

22     DomSetClust                                                                                                                            
 Gray

23     HGNN                                                                                                                               
 

24     DRCNN                                                                                                            
 Gray

25     View-GCN                                                                                                     
 

26     HMVCM                                                                                                          
 Gray

27     MVTN                                                                                                          
 28     MVT                                      
 Gray 29     OVPT                                                                               
 30     MVDAN                                                                                                      
 Gray 31     MVMSAN                                                                                

 32     Yu and Cao                                                                                 

 Gray33     VCGR-Net                                                                                 

 34     ViewFormer                                                             

 Gray 35     MVContrast                                                                                                  

  36     MORE                                                                                 
 
Gray 37     MVCVT                                                                                 
 





    



[p]



    
    0.5Different camera configurations (see Fig. <ref>) that have been used on the view-based 3D object recognition methods

max width=



No.     Multi-view Configurations:     (A)
 Circular     (B)
 Dodecahedral
Spherical     
(C)
 Random      
(D)
 Circular with
 Elevation      
(E) 
 Cylinder
Panoramic                            
                              
 
Gray 
1                      MVCNN                                                                                                                                                                                                        
 

2                      Pairwise                                                                                                                                                                                                       
 Gray

3                      VolumetricMVCNN                                                                                                                                 
 

4                     GIFT                                                                                                                                                                           
 Gray

5                      PANORAMA-NN                                                                                                                                                                       
 

6                      PANORAMA-ENN                                                                                                                                        
 Gray

7                     MHBN                                                                                                                                                                                                         
 

8                     GVCNN                                                                                                                                                                                                        
 Gray

9                     MVCNN(GoogLeNet)                                                                                                                                                                                  
 

10                     SeqViews2SeqLabels                                                                                                                                                                                           
 Gray

11                    RotationNet                                                                                                                
 

12                    Ma et al.                                                                                               
 Gray

13                     VERAM                                                                                                                                                                          
 

14                    MVCNN-new                                                                                                                                                                      
 Gray

15                     iMHL                                                                                                                                                                                                         
 

16                     MVSG-DNN                                                                                                                                                    
 Gray

17                   RN                                                                                                                                                                                                            
 

18                    MLVCNN                                                                                                                                                                                                       
 Gray

19                     G-CNN                                                                                                                                                                          
 

20                    DeepCCFV                                                                                                                                                                                                     
 Gray

21                     3D2SeqViews                                                                                                                                                                                                 
 

22                     DomSetClust                                                                                                                                                                                                 
 Gray

23                    HGNN                                                                                                                                                                                                         
 

24                     View-GCN                                                                                                                                    
 Gray

25                     HMVCM                                                                                                                                                                                                       
 26     MVTN                                                                                                                                                     
 Gray 27     MVT                                                                                                                    

28     OVPT                                                                                                                                    
 Gray
29     MVDAN                                                                                                                                                                                  

30     MVMSAN                                                                                                                        
  Gray 31     Yu and Cao                                                           

 32     VCGR-Net                                                                                                                        

 Gray
33     ViewFormer                                                                                                          

  34     MVContrast                                                                                                                          
 Gray 35     MORE                                                                                             

 36     MVCVT                                                                                                                        

 




[p]




    0.5Different numbers of the extracted views that have been experimented with the view-based 3D object recognition methods

max width=



No.      Number of Views:                  1     2     3     4     6    8     9     10     12     16     20     24     60     64     80  
 
Gray 1                      MVCNN                                                                                                                                                                                                                              
 

2                      Pairwise                                                                                                                                                                                                                                        
 Gray

3                      VolumetricMVCNN                                                                                                                                                                                                                                               
 

4                     GIFT                                                                                                                                                                                                                                                                   
 Gray

5                      PANORAMA-NN                                                                                                                                                                                                                                                               
 

6                      PANORAMA-ENN                                                                                                                                                                                                                                                            
 Gray

7                     MHBN                                                                                                                                                                                                                                         
 

8                     GVCNN                                                                                                                                                                                                                                                        
 
Gray
9                     MVCNN(GoogLeNet)                                                                                                                                                                                                                                  
 

10                     SeqViews2SeqLabels                                                                                                                                                                                                                  
 Gray

11                    RotationNet                                                                                                                                                                                                                                                 
 

12                    Ma et al                                                                                                                                                                                                                
 
Gray
13                     VERAM                                                                                                                                                                                                                                            
 

14                    MVCNN-new                                                                                                                                                                                                                                                                 
 
Gray
15                     iMHL                                                                                                                                                                                                                                                                     
 

16                     MVSG-DNN                                                                                                                                                            
 
Gray
17                   RN                                                                                                                                                                                                                                                
 

18                    MLVCNN                                                                                                                                                                                                                    
 
Gray
19                     G-CNN                                                                                                                                                                                                                                         
 

20                    DeepCCFV                                                                                                                                                                                                            
 
Gray
21                     3D2SeqViews                                                                                                                                                                                                                     
 

22                     DomSetClust                                                                                                                                                                                                                      
 
Gray
23                    HGNN                                                                                                                                                                                                                                                                       
 

24     DRCNN                                                                                                                                                                                                                                             
 
Gray
25                     View-GCN                                                                                                                                                                                                                                                  
 

26                     HMVCM                                                                                                                                                                                                                                                                     
 
Gray
27                    MVTN                                                                                                                                                                                                                                              

28                    MVT                                                                                                                                                                                                                    
 Gray
29     OVPT                                                                                                                                                                                            
 
30     MVDAN                                                                                                                                                                                                                    
Gray
31     MVMSAN                                                                                                                                                                                                           

   32     Yu and Cao                                                                                                                                                                                                                                    
 Gray 33     VCGR-Net                                                                                                                                                                                                                        
 
34     ViewFormer                                                                                                                                                                                   
 Gray 35     MVContrast                                                                                                                                                                                                                          

 36     MORE                                                                                                                                                                                                                                             
 Gray
 37     MVCVT                                                                                                                                                                                                                                                









  §.§.§ Views Selection Strategies



A key challenge for the 3D object recognition methods is to use the features extracted by the deep network and implement a feature fusion to produce a differentiated global descriptor for the object that will be used later to gain accurate classification. Feature fusion is the process of combining multiple views' features into a discriminated global descriptor. The original 3D object recognition methods utilized all views captured from all viewpoints for feature fusion <cit.>. However, because only some views are practical for recognition, some views are more discriminative, and processing all views needs heavy computation and causes overhead. Therefore, a more robust view selection mechanism is needed <cit.>. Therefore, some 3D object recognition methods presented view selection techniques to remove irrelevant and redundant views. If a view or its feature is poorly related to the class information, it is considered irrelevant. If two views or their features have an almost similar distribution, one is redundant <cit.>. The selected views are then regarded as qualified samples for training to gain a better feature set, achieving better performance. According to <cit.>, the different selection strategies of existing feature fusion methods can be divided into Passive Views Selection (PVS), Active Views Selection (AVS), and Active Camera Viewpoints Selection (ACVS) strategies. 

The PVS strategy uses feature selection techniques after involving all the captured views. So, the views are captured after setting all viewpoints earlier and then feeding all these views into the network. However, the PVS strategy is practical and straightforward, but it is computationally expensive and only accomplishes the best performance if it uses all views. Some techniques of this strategy are the max-pooling, the view-pooling, and the maxout layers <cit.>. MVCNN <cit.> and GVCNN <cit.> were using this type of selection. <cit.> used the first CNN of the MVCNN model to generate the features of each captured view and then aggregated all the extracted features (descriptors) in the view-pooling layer to merge them into one general 3D shape descriptor that the second CNN of the network uses for 3D object recognition and sketch recognition tasks. Later, <cit.> enhanced the object centering and switched to a black rendering background, which helped improve MVCNN performance. However, the same selection strategy on feature fusion was used and not improved <cit.>.

Similarly, <cit.> employed the same selection strategy in the view-pooling layer of the GVCNN model but in two steps. First, with GVCNN pooling, all the views within the same group are pooled to merge into a group-level descriptor. Then, these group-level descriptions are fused later to produce the final shape-level 3D object recognition. Yet, the PVS strategies depend on pooling, which only selects the average or maximum descriptor of all views. Moreover, according to <cit.>, it has a permutation invariance, which means it pays no attention to approximately all views’ content and spatial information between them.








The ACVS strategy does not use all views collected from all viewpoints because not every view is necessary for the network. So, it uses only a subset of views captured from specific viewpoints to feed them into the network. This strategy uses as few views as possible to reduce the computational cost and tackle the occlusion problem, which is significant for real-world scenarios and applications <cit.>. RotationNet <cit.> and MVTN <cit.> are differentiable and use this selection strategy. The idea of the RotationNet method <cit.> is that it will find the best pose (viewpoint variables) for each category label through unsupervised learning; hence, viewpoint variables will be dealt with as latent variables that are optimized in the training phase and then predicted with the category on the testing phase. On the other hand, MVTN <cit.> tackles the drawback of the previously proposed methods that rely on a fixed set of captured views using fixed camera viewpoints for all 3D objects in the datasets. Rather than the previous approaches where their scene parameters are pre-defined and selected for the entire 3D dataset using the most common camera viewpoints configurations which are circular and spherical around the object, MVTN has been proposed as an adaptive approach to learning viewpoints per object.






    


On the other hand, the AVS strategy also does not consider every view for recognition, but even so, it needs to feed all views captured from all viewpoints into the network. Then, it automatically selects the best views using the network to improve performance and prevent the computational cost. AVS is not only suffering from the same drawbacks but also for solving practical problems (e.g., occlusion); it is complex <cit.>. <cit.> proposed view-GCN for 3D object recognition based on the view selection idea. View-GCN implemented the widely used Farthest Point Sampling (FPS) in GCN <cit.> and then utilize a selective view-sampling strategy to sample representative views by view-selectors to build the next level view-graph. View-selectors are components, each learning to select the discriminative view from a set of local neighboring views. All the learned features in different levels are combined by performing max-pooling on the updated node features to be a pooled descriptor, and then the concatenation of all the pooled features at all levels is the final global shape descriptor. Despite the excellent performance of view-GCN, it still needs to feed the network with all views captured from all viewpoints. Also, OVPT <cit.> used this type of technique with the help of information entropy to evaluate view quality. Again, based on high information entropy values,  the best-view prediction model by MORE <cit.> trained to provide a set of best-selected views. Moreover, VCGR-Net <cit.> applied this type of selection through two stages. In the first stage, the model highlighted the discriminative region (features) using a spatial attention mechanism to decrease the redundant information between views. In the second stage, it removed the untrustworthy features using non-local neural networks by calculating the connections between all possible features across and within views. 



  §.§.§ Pre-trained CNN Architectures


CNN-based models have been utilize in computer vision and image processing problems. Hence, there are multiple architectures and variants of the basic CNN <cit.>. Consequently, the multi-view 3D object recognition methods take advantage of the CNN architectures pre-trained by the excellent publicly available large-scale 2D datasets such as ImageNet <cit.> and directly use them as a backbone network. So, the backbone network is a CNN architecture that has performed well in 2D classification tasks. The two fundamental roles of this pre-trained CNN backbone in the 3D recognition task are to extract the features from 2D rendered views at the beginning of the recognition model and to use the fused global shape descriptor for classification at the end of the recognition model. Using these CNNs as the backbone networks will save training time, reduce the complicated 3D recognition task to a simplified 2D recognition task, and improve the classification accuracy <cit.>. The most used CNN architectures as backbone networks on the multi-view 3D object recognition methods <cit.> are summarized in Table <ref>. It has been noticed from these related works that AlexNet <cit.>, VGG <cit.>, GoogLeNet <cit.>, and ResNet <cit.> architectures and their variants are the most popular networks that are employed as the backbone in this field. Therefore, Table <ref> summarizes these CNN variants in terms of their released year, main role or importance, number of parameters, and their advantages and disadvantages as listed by <cit.>.

[p]




    0.5Different CNN architectures that have been (: chosen) or (: just experimented) as backbones on the related works

max width=



No.     CNN architecture:               Noval CNN     VGG-M 
    VGG-C,D
    VGG-A     VGG-E    GoogLe-Net     AlexNet      ResNet-18    ResNet-34     ResNet-50     ResNet-152 
 
Gray 1                      MVCNN                                                                                                                                                                                                                                                                                                   
 

2                      Pairwise                                                                                                                                                                                                                                                                                                                          
Gray 

3                      VolumetricMVCNN
                                                                                                                                                                                                                                                  
 

4                     GIFT                                                                                                                                                                                                                                                                                                                      
 Gray

5                      PANORAMA-NN                                                                                                                                                                                                                                                                                                                
 

6                      PANORAMA-ENN                                                                                                                                                                                                                                                                                                           
 Gray

7                     MHBN                                                                                                                                                                                                                                                                                                                          
 

8                     GVCNN                                                                                                                                                                                                                                                                                                                      
 
Gray
9                     MVCNN(GoogLeNet)
                                                                                                                                                                                                                                                                                               
 

10     SeqViews2SeqLabels                                                                                                                                                                                                                                                                                                                  
 
Gray
11     RotationNet                                                                                                                                                                                                                             
 

12                    Ma et al.                                                                                                                                                                                                                                                                                               
 
Gray
13                     VERAM                                                                                                                                                                                                                                                
 

14                    MVCNN-new                                                                                                                                                                                                                                             
 
Gray
15     MVSG-DNN                                                                                                                                                                                                                                                                
 

16     RN                                                                                                                                                                                                                                                                                                                                        
 
Gray
17     MLVCNN                                                                                                                                                                                                                                                                                                                         
 

18     G-CNN                                                                                                                                                                                                                                                                                                                   
 
Gray
19     DeepCCFV                                                                                                                                                                                                                                                                                                      
 

20     3D2SeqViews                                                                                                                                                                                                                                                                        
 
Gray
21     DomSetClust                                                                                                                                                                                                                                                                                                                     
 

22     DRCNN                                                                                                                                                                                                
 
Gray
23     View-GCN                                                                                                                                                                                                                            
 

24     HMVCM                                                                                                                                                                                                                         
 Gray 25     MVTN                                                                                                                                                                                                                                                                                  

26     MVT                                                                                                                                                                                                                                                                                                           
 Gray
27     OVPT                                                                                                                                                                                                                                                                                           
 
28     MVDAN                                                                                                                                                                                                                                                                 
 Gray
29     MVMSAN                                                                                                                                                                                                                                                                   
 30     VCGR-Net                                                                                                                                                                                                                                                                                     
 Gray 31     ViewFormer                                                                                                                                                                                          
  

32     MVContrast                                                                                                                                                                                                                                                                                
 Gray
33     MORE                                                                                                                                                                                                                                                                                       
 34     MVCVT                                                                                                                                                                                                                                     
 










[p]




    0.5Summary of various CNN architectures  with their advantages and disadvantages (ER is the error rate)

max width=




CNN 
Architecture                            Dataset                                                      Importance                                                  Parameters     Advantages            Disadvantages                              
 
Gray AlexNet  <cit.>       ImageNet ILSVRC 2012 (ER= 16.4)     - Wider and deeper compared to standard CNN - Used dropout, RELU activation function, and overlap pooling     60 M     - Can extract high, mid, and low-level features using small and large size filters on the early and final layers -It introduced CNN regularization -It was first to use parallel GPUs as an accelerator to handle the complex architectures     - It contains dormant neurons in the first two layers - Extreme filter size which leads to undersampling of the trained feature maps 
  


VGG  <cit.>               ImageNet ILSVRC 2012 (ER= 7.3)    - It had a homogeneous topology  - It used kernels with a small size                138 M                       - It introduced the effective receptive field concept - It introduced the homogeneous and simple topology concept  - It is available in different configurations based on its deep layers: VGG-M (8 layers), VGG-A (11 layers), VGG-B (13 layers), VGG-C (16 layers), VGG-D (16 layers), VGG-E (19 layers)     -
  It uses fully linked layers which are computationally expensive                                                                                                                                                  
 
Gray
GoogLeNet  <cit.>     ImageNet ILSVRC 2014 (ER= 6.7)     - It was the first to propose the block concept - It used the idea of split transform and before merge          4 M                         - It introduced the multi-scale filters utilizing concepts within the layers - It introduced the divide, convert, and merge concept  - It enhances convergence rate using auxiliary classifiers                                    - It uses heterogeneous topology which makes parameter tuning difficult - Valuable information may be lost as a result of a representational bottleneck                     
 

ResNet  <cit.>             ImageNet ILSVRC 2015 (ER= 3.6)     Identified
  residual learning with mapping-based skip connections                                                                                           1.7
  M to 25.6 M             - It reduced the error rate for deeper networks - It introduced the residual learning concept - It alleviated the vanishing gradient problem effect                                                                                              - It is complicated - In feed forwarding, it reduced the feature-map information - Because of module stacking, it over-adapts the hyper-parameters for a particular job 









  §.§.§ Fusion Strategies


The most vital information for any successful recognition task is the features. However, if the features are confusing, overlapping, noisy, or distorted, they will constrain any classifier's performance. Hence, selecting a feature set is vital for any successful single or multiple classifier(s) <cit.>. In the view-based methods, when the CNNs extract the view-level features, feature fusion is required to create a global shape descriptor to provide an accurate object recognition <cit.>. Feature fusion aggregates multiple view-level features for information synthesized from all views into a single shape descriptor <cit.>. An early study in 2021 conducted by Seeland and Mäder <cit.> investigated the three fusion strategies utilize in DL-based multi-view recognition models. These strategies are early, late, and score fusions as shown in Fig. <ref>.



In the early fusion strategy, the convolutional feature maps are stacked after they are produced from the multiple CNNs and then subsequently processed together. In this strategy, the depth of the stacked feature maps needs to be reduced to a single-view feature map. There are two different approaches for depth reduction: i) early fusion (max), where a max-pooling operation is performed on the stacked feature maps, and ii) early fusion (conv), where a 1×1 convolution operation is performed instead of the max-pooling operation <cit.>.
On the other hand, the late fusion strategy depends on gathering the output of the last layer before the layer that does classification. There are two approaches to applying the late fusion strategy, which are: i) late fusion (max), where max-pooling operation is performed to the stacked feature vectors, and ii) late fusion (fc), where concatenation and fully connected fusion operation are performed <cit.>.
In contrast, the score fusion strategy is operated by element-wise aggregation of the softmax classification scores per image view. Where one of the following aggregation functions is performed: i) sum-score fusion where the scores of the views are summation, ii) product-score fusion where the scores of the views are multiplication, and iii) max-score fusion where the scores are maximum across views <cit.>.



  §.§.§ Evaluation Metrics

Typically, the evaluation of 3D object recognition methods encompasses two distinct categories of tasks, namely 3D classification and retrieval tasks.
 

    
  * 3D object classification: test the proposed model on a 3D object classification task which involves classifying a given object by identifying its category (a.k.a. class label) as shown in Fig. <ref>.a.  Two metrics have been used as criteria for object classification accuracy <cit.> to evaluate the classification performance of any 3D object recognition model:


    
  * OverallInstance accuracy (OA): also called average instance accuracy or instance-level accuracy. Which is the objectssamples that classified correctly to the total number of testing objectssamples <cit.>. This accuracy is intuitive since it reflects the number of the wrongly classified objectssamples <cit.>. Instance accuracy can be calculated using Equation <ref>.
    
    
    Instance Accuracy= ∑_i=1^CTP_i+TN_i/∑_i=1^CP_i+N_i

    

    
    
  * AverageClass Accuracy (AA): a.k.a. average per-class accuracy or class-level accuracy. Which is the mean or average of all the correctly classified class accuracy <cit.>. In other words, it is the mean of the instance accuracy among all classes. This accuracy is objective since the number of the tested objects from different categories is unequal <cit.>. Class accuracy can be calculated using Equation <ref> <cit.>.
    
        
    Class Accuracy= 1/C∑_i=1^CTP_i+TN_i/P_i+N_i

    


Where: 
C is the total number of experimented classescategories P and N are the numbers of positive and negative experimented samples, respectively. TP and TN are the numbers of true positive and true negative experimented samples, respectively. i is the corresponding category.
    
    
  * 3D object retrieval: test the proposed model on a 3D object retrieval task, which involves finding all the closest 3D objects in the dataset that match a given querying 3D object (having the same category as in Fig. <ref>.b). To evaluate the retrieval performance of any 3D object retrieval model, the mean average precision has been utilize as criteria for object retrieval accuracy <cit.>:
    
    

  
  * Mean Average Precision (mAP): which is the average value of Average Precision (AP) in each category (k) and can be calculated using Equation <ref>. AP is defined in Equation <ref> as the area under the PR curve (P as in Equation <ref> is the precision value, and Δr(k) is the recall value change from k-1 to k) <cit.>.
    
    
    mAP= ∑_k=1^CAP(k)/C


    
    AP=∑_k=1^NPrecision(k)Δ r(k)

    
    
    Precision =TP/TP+FN


    
    
    




  §.§.§ Recognition Performance


Several deep networks have been investigated for 3D object classification and retrieval. Among these networks, the view-based networks have performed best so far to achieve the current state-of-the-art performance <cit.>. Table <ref> summarizes the classification and retrieval performance of the view-based networks on the well known ModelNet40 and ModelNet10 datasets. 

From the classification performance of the view-based models, it can be seen that the best ModelNet40 results obtained by RotationNet <cit.>, OVPT <cit.>, MVT <cit.>, View-GCN <cit.>, and ViewFormer <cit.> with 97.37%, 97.48%, 97.50%, 97.60%, and 98.8% OA, respectively, where all of them were using 20 views with spherical configuration and a selection mechanism except ViewFormer <cit.> that employed random configuration. Note that OVPT <cit.> starts with 20 views and then is reduced to 6. On the other hand, from the ModelNet10 results, it can be noticed that the excellent classification performance obtained by ViewFormer <cit.>, MVT <cit.>, DRCNN <cit.>, and OVPT  <cit.>, with 99.30%, 99.30%, 99.34%, and 99.57% OA, respectively. However, they were using different numbers of views. 

Further, it is evident from the retrieval performance analysis that DRCNN <cit.>, G-CNN <cit.> and <cit.> achieved the highest results on both datasets, with 93.90%, 93.56%, and 95.98% mAP on ModelNet40, and with 96.15%, 96.18%, and 98.23% mAP on ModelNet10, respectively. The DRCNN <cit.> and <cit.> utilized a spherical configuration with 20 views, whereas the G-CNN <cit.> employed a circular configuration with 12 views.

It is worth mentioning that increasing the number of training samples will improve the recognition performance, as proven by  <cit.> where they trained their MORE model with a random fraction (full, third, and twentieth) of the ModelNet10 and ModelNet40 datasets separately to test whether subsets of it would achieve similar performance while minimizing the required training time. Finally, they show that increasing the training data size will lead to an improvement in performance.

Even more, two different training configurations affect the recognition performance, which is using i) the pre-trained backbone network on ImageNet images <cit.> as it is, or ii) fine-tuned the pre-trained backbone network on ModelNet dataset <cit.>. The two configurations have been experimented with by MVCNN <cit.> and GVCNN <cit.>, where both of them proven that fine-tuning significantly improves the recognition performance by a margin of +6.2%. Hence, the later proposed methods, such as view-GCN <cit.> and RotationNet <cit.>, fine-tuned the weights of the backbone network. However, some work, such as MVTN <cit.> and VERAM <cit.>, refrain from fine-tuning the weights of the backbone network to avoid the slow training progress due to the forward and backward propagation needed in each step.

[p]




    0.3Performance summary (OA: Overall Accuracy, and AA: Average Accuracy) of different view-based 3D object recognition methods achieved on the ModelNet40 <cit.> and ModelNet10 <cit.> datasets

max width=

 


No.      Method Name      Year     2*# of
 Views    3cModelNet40     
   3cModelNet10 


 
5-11
                  AA     OA     mAP         AA     OA     mAP


Gray
1                      MVCNN                          2015                                                                           12                                                                                    89.50%                                                        -      80.20%          -      -                                                                                            -                                                         
  

2                      Pairwise                         2016                                                                           12                                                                                          -      90.70%      -          -      94%                                                             -                                                               
 Gray

3                      VolumetricMVCNN        2016                                                                           20                                                                                    89.70%                                                        92%                                                    -              -      -        -                                                               
 

4                     GIFT                           2016                                                                           64                                                                                    89.50%                                                        -                                                   81.94%                91.50%       -      91.12%                                                             
 Gray

5                      PANORAMA-NN                        2017                                                                           6                                                                                     -                                                              90.70%                                                    -           -      91.12%        -                                                               
 

6                      PANORAMA-ENN                    2018                                                                           1                                                                                     -                                                              95.56%                                                 86.34%            -      96.85%           93.28%                                                         
 Gray

7                     MHBN                         2018                                                                           6                                                                                     93.10%                                                        94.70%                                                    -         95%      95%         -                                                               
 

8                     GVCNN                           2018                                                                           8                                                                                     90.70%                                                        93.10%                                                        84.50%          -      -     -                                                         
 
Gray
9                     MVCNN(GoogLeNet)      2018                                                                           12                                                                                    -                                                              92.20%                                                 83%           -      -         -                                                            
 
10                     SeqViews2SeqLabels               2018                                                                           12                                                                                    91.10%                                                        93.30%                                89%           94.80%      94.82%      91.43%                                                                              
 Gray

11                    RotationNet                   2018                                                                           20                                                                                    96.29%                                                        97.37%                                                     -         -      98.90%       -                                                               
 

12                    Ma et al.     2018                                                                           12                                                                                     -                                                              91.05%                                                  -          -      95.30%           -                                                               
 Gray

13                     VERAM                        2018                                                                           9                                                                                     92.10%                                                        93.70%                                              -            95.30%      95.50%             -                                                               
 

14                    MVCNN-new                       2018                                                                           6                                                                                     92.40%                                                        95%      -           -      -                                                       -                                                               
 
Gray
15                     iMHL                        2018                                                                           12                                                                                     -                                                              97.20%                                                        -         -       -      -                                                               
 

16                     MVSG-DNN                        2019                                                                           12                                                                                    92.30%                                                        -                                                        83.70%          94%      -          93.90%                                                         
 
Gray
17                   RN                            2019                                                                           12                                                                                    92.30%                                                        94.30%                                                    86.70%          95.10%      95.30%     -                                                              
 

18                    MLVCNN                          2019                                                                           12                                                                                    94.16%                                                        92.84%                                               92.84%             -       -      -                                                            
 
Gray
19                     G-CNN                            2019                                                                           20                                                                                    92.60%                                                        94.67%                                                    93.56%           -      96.78%      96.18%                                                          
 

20                    DeepCCFV                         2019                                                                           12                                                                                     -                                                              92.49%                                            -           -      -               -                                                                
 
Gray
21                     3D2SeqViews                     2019                                                                           12                                                                                     91.50%                                                        93.40%      -         -      -                                                        92.12%                                                               
 

22                     DomSetClust                 2019                                                                           12                                                                                     92.80%                                                        93.80%                                                   -         -            -      -                                                             
 
Gray
23                    HGNN                             2019                                                                           12                                                                                     -                                                              96.70%                                                    -          -      -        -                                                               
 

24                    DRCNN                      2020                                                                           12                                                                                    94.86%                                                        96.84%                                                   93.90%         99.30%      99.34%         96.15%                                                         
 
Gray
25                     View-GCN                          2020                                                                           20                                                                                    96.50%                                                        97.60%                                          -          -     -         -                                                     
 

26                     HMVCM                           2021                                                                           12                                                                                    94.57%                                                        92.80%                                            92.80%          95.7%      -          93.90%                                             
 Gray
27                    MVTN                            2021                                                                           20                                                                                    92.20%                                                        93.50%                                                    92.90%            -      -     -                                                          

28                    MVT                            2021                                                                           20                                                                                    -                                                        97.50%                                                 -            -      99.30%          -                                                         
 Gray 29                    OVPT                            2022                                                                           6                                                                                      96.74%                                                        97.48%                                                   -           99.21%      99.33%       -                                                         
 30                    MVDAN                           2022                                                                           12                                                                                      95.50%                                                        96.60%                                                -             -      -        -                                                         
 Gray 31                    MVMSAN                           2022                                                                           20                                                                                      95.68%                                                        96.96%                                                 -          98.42%      98.57%          -                                                         
 32     Yu and Cao     2023                                                                           20                                                                                      -                                                        97.13%                                                 95.98%          -      99%          98.23%                                                         
 Gray 

33                    VCGR-Net                        2023                                                                           12                                                                                     93.33%                                                        95.62%      -           97.24%      97.79%                                                       -                                                               
 34                   ViewFormer                            2023                                                                           20                                                                                      98.90%                                                        98.80%                                                 -          100%      99.30%          -                                                         
 Gray 

35                    MVContrast                        2023                                                                           12                                                                                     90.24%       92.54%      90.30%           -      -                                                       -                                                               
36                    MORE                            2023                                                                           1                                                                                      -                                                        97.16%                                                 -          -      98.26%          -                                                         
  Gray 

37                    MVCVT                        2023                                                                           12                                                                                    -                                                        95.40%       -           -      98.68%                                                       -                                                               











§ TRANSFORMER-BASED MULTI-VIEW 3D OBJECT RECOGNITION


Transformer is an attention-based block that simulates a human visual system by not processing an entire scene; instead, it focuses on salient parts selectively in order to improve the capturing of visual structure and better represent the interests <cit.>. Transformers have proved recently their excellent performance on a broad range of Natural Language Processing (NLP) applications such as machine translation, text classification, and question answering. This attracts the interest to adapt these models in the computer vision field, especially for multi-learning tasks <cit.>. One of these multi-learning tasks is the multi-view 3D object recognition that benefits from different transformer-based models such as MVT <cit.>, MVDAN <cit.>, OVPT <cit.>, MVMSAN <cit.>, MVCVT <cit.>, and <cit.>.


Multi-view DL-based models that used view-based pooling such as MVCNN <cit.> and  GVCNN <cit.>, or patch-based pooling such as View-GCN <cit.> did not have visual features interactions between multiple views in previous layers but only apply visual features fusion from multiple views in the last pooling layer. This configuration limits its effectiveness and fails to distinguish patches from different views. Hence, this problem inspired <cit.> to propose a Multi-view Vision Transformer (MVT) for 3D object classification. MVT involves two parts. The first part splits each view individually into non-overlap patches, and for patches in each, it generates the low-level features view using a local transformer encoder. The second part concatenates the low-level patch features from all views as a global set and then feeds the concatenated set into the global transformer encoder to communicate the visual features in patches from multiple views. After that, the output patch features are sum-pooled into a global descriptor that feeds the MLP head for classification. MVT has experimented with the two camera settings, 12-view and 20-view, on the two 3D datasets: ModelNet40 and ModelNet10. MVT outperformed other transformer-based approaches such as OVPT <cit.>, MVDAN <cit.>, and MVMSAN <cit.> with 97.5% OA On ModelNet40 and with 99.3% OA On ModelNet10 using 20-view camera setting.

Some of the existing multi-view 3D object recognition methods ignore the views' distinguishability and the hierarchical correlation between them. Hence, <cit.> proposed a Multi-View Dual Attention Network (MVDAN) for the task of 3D object classification. MVDAN contains two basic blocks: View Channel Attention Block (VCAB) and View Space Attention Block (VSAB). VCAB focuses on the critical views in the same category by finding the relationship between multiple views and highlighting important perspectives. In comparison, VSAB focuses on the more discriminatory details between the views of different categories. MVDAN was evaluated with a circular camera configuration of 12 views on ModelNet40 objects and Resnet50 as the backbone network. The MVDAN method achieves 96.6% OA and 95.5% AA while reaching 96.1% and 96.3% AA when experimented with 3 and 6 views, respectively.






Another work of multi-view 3D object classification that tackles the correlation between views is the Optimal Viewset Pooling Transformer (OVPT) proposed by <cit.>. OVPT aims to improve the recognition performance by reducing the views’ redundancy. OVPT has three modules; the first module captured the 20 spherical views from each 3D object and utilized information entropy as a selection mechanism to reduce the redundant views and gain the optimal view set. Then, the second module inputs the optimal views to pre-trained resnet34 as the backbone network for feature extraction that is later flattened into a local view token sequence so it can be input to the transformer. Finally, the last module is the pooling transformer that generates the global descriptors used for classification. OVPT has been evaluated on both ModelNet10 and ModelNet40. As a result, it achieves 97.48% OA and 96.74% AA on ModelNet40 with only six best views, and it achieves 99.33% OA and 99.21% AA on ModelNet10 with nine best views. Compared with other DL-based methods such as iMHL <cit.> and RotationNet <cit.> and other transformer-based methods such as MVDAN <cit.> and MVMSAN <cit.>, OVPT achieves state-of-the-art performance in the multi-view 3D object classification accuracy with less redundant views and less computational resources.

In order to improve the classification accuracy by solving the problems of poor generalization ability and insufficient extracted features of the existing multi-view 3D network model, <cit.> have proposed a Multi-View SoftPool Attention convolutional Network framework (MVMSAN) for the 3D classification task. The MVMSAN model has three parts; the first part uses ResNest as a backbone network and adaptive average pooling module to extract multi-view features from 20 spherical views captured from each object. The second part used the extracted multi-view feature as the Query for their proposed SoftPool attention convolution method to extract refined view feature information during downsampling and calculate attention scores by Query and Key that feed to the last part of the model. Finally, the last part fused refined view features and generated a 3D global descriptor for classification. MVMSAN was evaluated on both ModelNet40 and ModelNet10. MVMSAN outperformed some deep-based multi-view 3D object recognition methods such as MVCNN <cit.>, MHBN <cit.>, VERAM <cit.>, and RN <cit.>. Chosen ResNest14d as their backbone network, they achieved on ModelNet40 96.96% OA and 95.68% AA, and on ModelNet10 98.57% OA and 98.42% AA.

More recently, another transformer-based works <cit.> were proposed to handle the problem of ignoring the correlation and connection between the multi-views that suffered by most traditional deep-based multi-view 3D object recognition methods such as <cit.>. <cit.> proposed a view self-attention network model for 3D object classification and retrieval. The self-attention approach solves the problem of input order in the sequence views by calculating the correlation between them and focusing only on the important views to extract the global information of the object better. To apply this approach, Yu and Cao’s model consists of two units: i) the single-view self-attention network, which discovers the connection between features of a single view, and ii) the multi-view self-attention network, which discovers the relationship between the different views to reduce information redundancy by finding the most critical views. They validate their model with the ModelNet40 and ModelNet10 datasets with 12-view and 20-view camera settings. On ModelNet40, their method achieved 95.30% OA and 92.26% mAP with 12-views and 97.13% OA and 95.98% mAP with 20-views. While on ModelNet10, their method achieved 97.23% OA and 95.92% mAP with 12-views and 99% OA and 98.23% mAP with 20-views. In the same year, ViewFormer is proposed by <cit.>. to handle the limitation of different ways when multi-view are aggregated, such as when the aggregated views are insufficient (e. g., MVCNN <cit.>), collected from sequence around the 3D object which is invalid assumption (e. g., SeqViews2SeqLabels <cit.>), or when introducing additional computational overheads using graph construction e.g., view-GCN <cit.>). ViewFormer, a view set attention model for multi-view 3D object classification and retrieval,  presented a View Set perspective where several views of a 3D object are organized, which makes the elements’ permutation invariant. ViewFormer has four units: i) Initializer is AlexNet to initialize the views representations, ii) Encoder is modified from the standard Transformer encoder, iii) Transition summarizes the learned higher-order correlations to View-Former’s understanding and expresses the 3D object. iv) The decoder is designed for classification and retrieval tasks. The ViewFormer is evaluated on the three classification datasets: ModelNet10, ModelNet40, and RGB-D, and the ShapeNet Core55 as the retrieval dataset. ViewFormer used random viewpoints of the views with random numbers; however, 20 views reached the optimal performance. Compared to attention-based methods such as VERAM <cit.>, SeqViews2SeqLabels <cit.>, and 3D2SeqViews <cit.>, these methods do not yield acceptable results since the highest accuracy on ModelNet40 is 93.7% OA, whereas ViewFormer reaches 98.9% AA and 98.8% OA. Additionally, compared to transformer-based methods such as MVT <cit.> on ModelNet10, ViewFormer shows that taking the patch-level interactions into account is unnecessary to accomplish the 99.30% OA. Later on, Multi-View Convolutional Vision Transformer (MVCVT) <cit.> was proposed for 3D object classification to tackle the challenge of forming a represented global shape descriptor since MVT <cit.> uses a transformer for feature extraction which is computationally expensive and ignores the specifics of the local features. MVCVT was designed as a hybrid CNN-transformer model where it used a GoogleNet as a backbone CNN for feature extraction of different views and then used a transformer for highlighting the feature relationship between these views. It used a circular camera setting to capture 12 views around the 3D object. When evaluated on ModelNet10 and ModelNet40, it achieved 98.68% and 95.4% OA, respectively, which is competitive to MVT <cit.> results.

It is worth mentioning that hybrid deep-transformer models such as MVT <cit.>, OVPT <cit.>, MVDAN <cit.>, MVMSAN <cit.>, and MVCVT <cit.>, that used both transformer and CNN for 3D object recognition are also applied most of the steps in the general pipeline in Section <ref>.  



§ RELEVANT COMPUTER VISION APPLICATIONS AND TASKS




The multi-view recognition methods can be extended to be generalized to various computer vision applications that are based on image classification such as plant species identification <cit.>, ant genera identification <cit.>, fingerprint identification <cit.>, hand-drawn sketch recognition <cit.>, car categorization <cit.>, breast cancer classification <cit.>, etc. The general multi-view classification methods used DL can be classified into two categories based on the computer vision application/domains it serves. The first category is the methods implemented on specific computer vision applications <cit.>, whereas the second category is those generalized to more than two different computer vision applications <cit.>.


When human specialists distinguish the 3D fingerprint, they identify the 3D fingerprint's global shape and local surface features from different viewpoints and angles. Inspired by that, multi-view DL approaches have been investigated for recognizing 3D fingerprints. Such approaches can offer more accurate 3D fingerprint recognition results. <cit.> present a method as the first attempt to investigate the problem of contactless and partial 3D fingerprint identification since it is more global and accurate personal identification than contact-based fingerprint recognition. Despite its benefits, partial 3D fingerprints often result from contactless 3D capturing, significantly reducing the matching accuracy. Hence, the authors proposed a fine-tuned multi-view CNN-based network with two channels to learn discriminative feature representation for solving this challenging problem. For training, three images containing two side-view and one top-view (as in Fig. <ref> are fed as inputs to the three Siamese networks. This trained multi-view CNN learns the 3D fingerprint concatenated features containing texture ridge-valley details, depth, and shape discriminative information.  
On the other hand, DL also has been broadly applied to medical screenings such as mammographic image classification. However, the current methods are mainly based on single-view mammography, which can't extract sufficient discriminative features, resulting in unacceptable classification accuracy. <cit.> solved this problem and proposed a novel multi-view mammography approach using CNN to classify the breast mass as benign or malignant. Using two mammographic views of each breast in their approach, they obtained breast mass complementary information that defines its shape, density, edge morphology, degree of calcification, and location. These two views, as shown in Fig. <ref> are mediolateral oblique (MLO), which is a lateral view and craniocaudal (CC) which is a top head-to-toe view. 




The implementation of multi-view classification techniques primarily focuses on particular applications within computer vision. However, <cit.> generalized their multi-view classification method to other applications. They present the first systematic investigation of the effectiveness of three different feature fusion strategies for multi-view classification in the machine learning area and generalize it to various computer vision applications. Specifically, these three fusion strategies are fused early, fused late, and fused with a score. The fusion strategies were generalized on fine-grained applications such as car categorization, plant identification, ant genera identification, and plant species identification. These classification applications were evaluated on four multi-view datasets: CompCars <cit.> (see Fig. <ref>.a) that contain five views (front, rear, side, front-side, or rear-side), PlantCLEF <cit.> (see Fig. <ref>.b)  that contains two views (flower and leaf), AntWeb <cit.> (see Fig. <ref>.c) that contain three views (head, profile view, and dorsal), and data captured using Flora Incognita <cit.> and Flora Capture <cit.> smartphone applications that contain two views (side-view of their flowers and top-view of their leaves). Moreover, in addition to 3D object recognition, <cit.> generalize their well-known MVCNN to be used for hand-drawn sketch classification and sketch-based shape retrieval with data jittering/augmentation to simulate the effect of views. They used data jittering to obtain additional samples (multi-view images) from a given 2D sketch. In their work, each hand-drawn sketch was in-plane rotated with three angles: -45, 0, 45, and also horizontally reflected to end up with six samples per 2D sketch image. 







All the above-proposed multi-view DL models <cit.> were capturing the multi-view of the object from different angles or orientations. On the other hand, <cit.> considered the views as the vertical partition of the extracted features from CNN using the random split technique. This work is regarded as the first to use feature set partitioning with CNN for different image classification applications. The Multi-view Convolutional Neural Network (MvCNN) used CNN for feature extraction from the inputted image. Then, they applied a random split for the view construction using the extracted features. 






Table <ref> summarizes multi-view DL models  <cit.> and the applied datasets with its computer vision application. The datasets that used for different multi-view computer vision applications with their references are shown in Table <ref> in Appendix <ref> for lack of space in Table <ref> and for easy access. It has been noticed that the multi-view datasets that were used for evaluation contained either 3D data, multi-view images, or 2D images with data augmentation to obtain multi-view images.

Even though the focus of this review is on multi-view 3D object recognition including 3D classification and 3D retrieval tasks, there have been some other important and relevant 3D tasks that are out of the scope of this review but worth exploring such as: multi-view 3D object detection <cit.> that used mainly by the recent autonomous driving system to detect 3D objects from different views and perceive the environment, multi-view 3D semantic scene segmentation <cit.> for understanding of visual data and improving the automated analysis of considerable 3D scenes in indoor environments, and multi-view 3D reconstruction <cit.> where 3D objects are represented from different views is essential task for virtual reality, 3D modeling, and computer animation.

[p]




    0.4Summary of some computer vision applications that used multi-view DL models for classification (ACC= Classification Accuracy, and 2D+Augment= 2D with data augmentation)



max width=
 




        
Refrence     Applications     Dataset Name         Dataset Type      Views     Classes      Samples     Metric       ACC

 
Gray     -Car categorization      CompCars (2015)                              Multi-view         5                        601                                                  8,183                      Top-1                                                96.72%       
 Gray


    -Plant identification                                                       PlantCLEF (2016)                                                                            Multi-view                                                                      2                        53                                                   1,839                             Top-1                                                                94.25%       
 Gray -42.4cm<cit.> 

     -Ant genera identification                                                    AntWeb (2021)                                                                               Multi-view                                                                      3                        82                                                   38,914                        Top-1                                                                    94.54%      
 Gray

      [l]-Plant
  species
 identification                                               Flora Incognita (2021)                                                                       Multi-view                                                                      2                        775                                                  8,557                          Top-1                                                                   96.07%       
 
22.4cm<cit.>     
23cm-Contactless and partial 3D fingerprint identification     Contactless 3D fingerprint Database (2008)                                                 3D        -                        336                                                  2016                   
    Top-1     99.89%       
 

              Multi-view Fingerprint Database (2014)                                           Multi-view                                                                      3                        1500     3000                         Top-1                                                                     85.40%       
 

Gray                           -3D object  recognition                                              ModelNet40 (2015)     3D                                                      80                       40                                                   12,311                        OA                                                                    90.10%  
   Gray   -32.4cm<cit.>          -Sketch classification                                                        SketchClean (2014)      2D+Augment                               6                        160                                                  8960                       OA                                                       87.20%        
Gray  

    -Sketch-based
  shape  retrieval                                               SketchClean
  (2014) + ModelNet40 (2015)     2D
  and 3D, respectively                                                        12                       10                                                   193                        mAP                                                                    36.10%        
 

22.4cm<cit.>        23cm-Breast cancer classification                                DDSM (2000)                                                                                  Multi-view                                                                      2                        2                                   1,445                      AA        82.20%        
 

          MIAS (1994)                  2D+Augment     2                         2                                                    322                         AA     63.06%       
 
Gray       -Digit recognition                                                            MNIST (1998)                                10                                                   70,000                     MA                                     96.80%        
 Gray

     -Gender classification           Men-Women Classification (2019)                                     2                                                    2893                             MA                                                               68.60%        
 Gray

     -Image  classification         CIFAR-10 (2009)                                                      10                                                   60,000                       MA                                                                   64.90%        
 Gray

     -Image classification                                                       CIFAR-100
  (2009)                                                                                                                                                                                  100                                                  60,000                                                         MA              45.60%        
 Gray  -52.4cm<cit.> 

      [l]-Fashion product 
classification               Fashion-MNIST (2017)                                                                                                                                                   -52cm2D with partition of extracted features                                -51cm2-6                   10                                                   60,000                              MA                                                              93.10%        
 Gray

     -Sign language  recognition                                                  American Sign Language (2019)                                                                                                               57                                                   2515                        MA                                                                      97.60%        
 Gray

     -Fruit recognition       Fruit 360 (2018)                                                                                                                                                       131                                                  67,692                      MA                                                                      82.00%         
 Gray

     -Tomato plant disease classification     Tomato Disease Classification (2019)                                                                                                                     10                                                   18,345                                                          MA                                  99.90%     
 Gray

      -Plant disease classification                                                 Plant Disease (2019)                                                                                                                                                                                15                                                   20,638                                                                            MA                84.90%       








§ FACTORS IMPACTING THE RECOGNITION PERFORMANCE AND FUTURE DIRECTIONS


To provide the readers with more ideas, several facts and future directions for the development of multi-view 3D object recognition methods are highlighted as follows:



  
Viewpoints and number of views
For multi-view methods, if the number of views from various perspectives is reduced, for example, to two, then the recognition performance will decrease due to the loss of much of the information that makes sense. On the other hand, if the number of views from various perspectives increases, then the recognition performance will improve and be much more stable <cit.>. Even more, using 20 views, the state-of-the-art performance was achieved by RotationNet <cit.>, view-GCN <cit.>, OVPT <cit.>, and ViewFormer<cit.>. In addition, when the model randomly selects the views, worse classification performance is achieved. If these selected views are randomly captured from a close or identical direction, this will significantly reduce the performance because it will have the same effect as using a few views <cit.>. According to <cit.>, methods to select the relevant number of views are an open research question.



  
Backbone and feature extracting
Including more complex or deeper backbones for feature extraction does not improve the performance <cit.>. However, performing fine-tuning on the network, as done by MVCNN <cit.>, RotationNet <cit.>, and View-GCN <cit.>, can significantly enhance the recognition performance. Hence, analyzing and comparing the effect of fine-tuning in different backbone architectures could be investigated more in the future.



  
Feature fusion and view selection
Most of the previously proposed methods did astonishing work; however, they fused features from a fixed set (e.g., 12, 20, or 80) of captured views using fixed camera viewpoints for all 3D objects in the datasets. The scene parameters are pre-defined and fixed for the entire 3D dataset <cit.>. The most common camera viewpoint configurations are circular and spherical around the object. Fusing all these views could confuse the classifier and be misleading for some classes (e.g., looking from the bottom at a bed). Moreover, in real-world circumference, and due to occlusions, objects can be observed from limited viewpoints and angles, making it hard to rely on multi-view images learned with the whole scenarios <cit.>. Even more, from the classification performance of the view-based models when they were evaluated on the well-known ModelNet40 dataset, the best results were obtained by RotationNet <cit.> and View-GCN <cit.>, which used a selection mechanism with 97.37% and 97.6% accuracy, respectively. The selection mechanism is used because not every view is useful for classification; some views are more discriminative for object classification, and processing all views needs a heavy computation <cit.>. What types and techniques are better for selection with a focus on view selection and qualification techniques is still open research to be investigated.



  
Backbone and feature extracting
Including more complex or deeper backbones for feature extraction does not improve the performance <cit.>. However, performing fine-tuning on the network, as done by MVCNN <cit.>, RotationNet <cit.>, and View-GCN <cit.>, can significantly enhance the recognition performance. Hence, analyzing and comparing the effect of fine-tuning in different backbone architectures could be investigated more in the future.



  
Feature fusion and view selection
Most of the previously proposed methods did astonishing work; however, they fused features from a fixed set (e.g., 12, 20, or 80) of captured views using fixed camera viewpoints for all 3D objects in the datasets. The scene parameters are pre-defined and fixed for the entire 3D dataset <cit.>. The most common camera viewpoint configurations are circular and spherical around the object. Fusing all these views could confuse the classifier and be misleading for some classes (e.g., looking from the bottom at a bed). Moreover, in real-world circumference, and due to occlusions, objects can be observed from limited viewpoints and angles, making it hard to rely on multi-view images learned with the whole scenarios <cit.>. Even more, from the classification performance of the view-based models when they were evaluated on the well-known ModelNet40 dataset, the best results were obtained by RotationNet <cit.> and View-GCN <cit.>, which used a selection mechanism with 97.37% and 97.6% accuracy, respectively. The selection mechanism is used because not every view is useful for classification; some are more discriminative for object classification; however, processing all of them needs a heavy computation <cit.>. What types and techniques are better for selection with a focus on views selection and qualification techniques is still open research to be investigated.



  
Light Direction and Object Color

<cit.> studied the impact of light's direction on the recognition performance and found that selecting a random light in the training phase benefits the network to generalize to the test set where it can reduce the overfitting for more significant views number. However, in the testing phase, relative light should be used to stabilize the performance. In addition, when they studied the impact of the objects' color on the recognition performance, they also found that using random objects' colors in the training phase can help to enhance the test accuracy. This fact could be investigated in more detail in future research.



  
Number of transformer blocks
If the multi-view 3D object recognition model is transformer-based, then the number of transformer blocks is another factor that could impact the recognition performance. The transformer-based models such as MVT <cit.>, OVPT <cit.>, and MVCVT <cit.> experimented with the influence of different numbers of transformer blocks. These methods agree that when the number of transformer blocks increases, it does not always lead to better recognition performance but will require more training time, more trainable parameters, and more floating-point operations. However, OVPT <cit.> and MVCVT <cit.> experimented on the ModelNet10 dataset with 0 to 6 gradually increased transformer blocks and concluded that three transformer blocks as an optimal choice with better accuracy and less computational cost. However, MVT <cit.> experimented on the same dataset with the same setting of transformer blocks and concluded that four were the optimal choice. This makes transformer block factors required to be investigated from various transformer-based models in the future.





§ CONCLUSION


The field of 3D data representation has experienced substantial growth in recent years, focusing on multi-view representations as the input for 3D object recognition methods due to their proven efficacy.  
The scope of this review is limited to DL-based and transformer-based methods, specifically those that use multi-view data as input representation. The paper provides a comprehensive overview of the general pipeline of DL-based multi-view 3D object recognition models, summarizing the various techniques utilized at each stage. Additionally, the article delves into the details of existing DL-based multi-view 3D object recognition models, including commonly used 3D datasets, camera configurations, view selection strategies, pre-trained CNN architectures, fusion strategies, and recognition performance. In addition, it contains the latest developments in transformers and attention mechanisms in multi-view 3D object recognition, which are not covered by the existing reviews. Finally, the paper highlights several key insights and future directions for developing multi-view 3D object recognition methods.










§ DECLARATIONS






  * Conflict of interest. The authors have no conflicts of interest to declare that are relevant to the content of this article.





















§ RELATED DEEP MODELS

The view-based deep models that proposed for 3D object recognition methods are shown in Table <ref> along with their references.



















§ MULTI-VIEW DATASETS

The datasets that used for different multi-view computer vision applications are shown in Table <ref> along with their references.


























