Recent Developments from Attribute Profiles for Remote Sensing Image Classification
Minh-Tan Pham Sébastien Lefèvre Erchan Aptoula Lorenzo Bruzzone



















Recent Developments from Attribute Profiles 
 for Remote Sensing Image Classification

    
Minh-Tan Pham
IRISA - Université Bretagne Sud 

UMR 6074, F-56000, Vannes, France




Erchan Aptoula
Institute of Information Technologies 

Gebze Technical University, 41400 Kocaeli, Turkey


Sébastien Lefèvre
IRISA - Université Bretagne Sud 

UMR 6074, F-56000, Vannes, France




Lorenzo Bruzzone
Dept. of Inf. Eng. Computer Science 

University of Trento, I-3812 Trento, Italy








    December 30, 2023
=========================================================================================================================================================================================================================================================================================================================================================================================================




Morphological attribute profiles (APs) are among the most effective methods to model the spatial and contextual information for the analysis of remote sensing images, especially for classification task. Since their first introduction to this field in early 2010's, many research studies have been contributed not only to exploit and adapt their use to different applications, but also to extend and improve their performance for better dealing with more complex data. In this paper, we revisit and discuss different developments and extensions from APs which have drawn significant attention from researchers in the past few years. These studies are analyzed and gathered based on the concept of multi-stage AP construction. In our experiments, a comparative study on classification results of two remote sensing data is provided in order to show their significant improvements compared to the originally proposed APs.



mathematical morphology, attribute profiles, multilevel image description, image classification, remote sensing






§ INTRODUCTION

Image classification is one of the most crucial tasks in remote sensing imagery which serves for several applications in land use and land cover mapping and monitoring. With the emergence of high resolution remote sensing technology, the exploitation of the spatial information together with the spectral characteristics becomes more and more significant to characterize and discriminate different thematic classes present from the image content. Within such spatial-spectral context, morphological profiles (MPs) <cit.> were extensively exploited during the 2000's <cit.> thanks to their multilevel analysis of spatial information by applying a sequence of opening and closing by reconstruction operators with increasing-size structuring elements (SEs). However, their high computation complexity prevent them to deal with large-size images. Besides, SEs can only model the size and scale of regions without their gray-level characteristics, thus not considering contextual features such as texture and contrast. 
 
To overcome the MPs' shortcomings, morphological attribute profiles (APs) <cit.> were proposed in early 2010's as their generalization and consist in applying a sequence of attribute filters (AFs) which are more powerful than operators by reconstruction. These AFs can decompose the image according to different types of attribute (i.e. any geometric and statistical features of regions), not only restricted to the scale and size of SEs employed by MPs. Besides, the construction of APs can be efficiently implemented based on the hierarchical representation of image via tree structures (i.e. originally via min- and max-tree <cit.>), hence better dealing with large-size remote sensing images. Scalability is further ensured with parallel implementations <cit.>.

In the past few years, a great number of research studies have been devoted to exploit and extend the use of APs applied to remote sensing image analysis, especially for classification task. These studies have been designed to improve the classification performance by focusing on the AP construction framework or adapting their use to different types of input data. In this paper, we conduct a survey on recent research studies that have been proposed and developed from the concept and application of APs. By decomposing the AP generation scheme into different stages, we regroup these studies into each specific stage in order to better provide an overview of their contribution to the general AP framework. We note that a recent survey <cit.> also exists but its contribution has focused only on the spatial-spectral approaches using different spectral feature extraction techniques and spatial processing by the standard APs <cit.>. Our survey involves more complete and detailed investigations of different developments and extensions from APs to improve their performance and optimize their construction framework. 


We first recall the background of APs and highlights the key components from AP construction framework (Sec. <ref>), before revisiting and discussing different developments from APs which have provided considerable contributions in the past few years (Sec. <ref>). An experimental study (Sec. <ref>) provides a comparative evaluation of some extensions compared to the original APs by conducting supervised classification experiments on two remote sensing image data. We finally conclude the paper and indicate future research directions (Sec. <ref>).



§ PRINCIPLE OF APS


APs are multilevel image description tools obtained by successively applying a set of morphological attribute filters (AFs) <cit.>. Unlike usual image filtering operators which are directly performed on pixel level, AFs work on connected component (CC) level based on the concept of image connectivity. In other words, an AF is a filtering operator applied on CCs with regard to a specific attribute characterizing the size, shape, or other properties of objects and regions present in the image. That is why AFs are more general than operators by reconstruction (which are limited to the size and shape of SEs), and why APs are more general than MPs <cit.>.

The generation of the standard APs <cit.> from an input image can be summarized as a four-step framework (see Fig. <ref>):



  * construct the hierarchical tree to represent the image. In <cit.>, the authors proposed to form one min-tree and one max-tree to encode the image;

  * compute some relevant attributes describing the geometrical and statistical features from each tree's node which corresponds to one connected component;

  * filter the tree by keeping/removing nodes according to their attribute values compared to predefined thresholds;

  * reconstruct the image from the filtered tree.

Steps 3) and 4) can be done for different attributes (with different threshold values) to finally produce a set of filtered images (by stacking them) forming the final APs.

More formally, given a grayscale image X:E →ℤ, E ⊆ℤ^2, the standard generation of APs on X is achieved by applying a sequence of AFs based on a min-tree (attribute thickening operators {ϕ_k}_k=1^K) and a max-tree (i.e. attribute thinning operators {γ_k}_k=1^K) as follows:

    (X) = {ϕ_K(X),ϕ_K-1(X),…,ϕ_1(X),X, 
    γ_1(X),…,γ_K-1(X),γ_K(X) },

where ϕ_k(X) is the filtered image obtained by applying the attribute thickening ϕ with regard to the threshold k. Similar explanation is made for γ_k(X). As observed, the resulted (X) is a stack of (2K+1) images including the original image, K filtered images from the thickening profiles and the other K from the thinning profiles. 



§ RECENT ADVANCES FROM APS


As described above, the construction of APs involves four main stages which are in fact the key components that have been focused for improvements by different literature studies within the past few years. In addition, since APs basically work on panchromatic images, some pre-processing or specific adaptation procedures are required when dealing with other input data (multi-channel images, radar data, etc.). Then, spatially post-processing the output profiles to increase their description capacity for classification has also drawn attention of researchers in several research studies. 
 
We now revisit the recently proposed developments that have provided significant contributions to adapt and improve the AP framework for remote sensing image classification (Sec. <ref> to Sec. <ref>). Here, our investigation will focus on three main key features: 


  * the adaptation of APs to other input data, in particularly to multi-channel images (Sec. <ref>);

  * the construction of APs using various tree representation structures (Sec. <ref>);

  * the AP post-processing using different feature enhancement techniques (<ref>).

For other related extensions (Sec. <ref> and Sec. <ref>), we provide standard concepts and refer readers to the related references for further details.



 §.§ Input data


Since APs were originally proposed to deal with panchromatic images <cit.>, their adaptation to other kinds of remote sensing data becomes quite significant. In particular, the application of APs to the classification of multi-channel images (multispectral and hyperspectral) has become one of the hottest research topics in this field. The idea is to perform a spatial-spectral approach for classification by combining rich spectral information from these data with efficient spatial modeling capacity of APs.

The standard extension of APs on hyperspectral images was proposed in <cit.> by first applying the principal component analysis (PCA) on the image and then extracting APs from some first principal components. The advantage of PCA is that this low-complexity technique can compress most spectral information from the hyperspectral image into only some first principal components. Hence, applying APs on these components may perform a basic spectral-spatial feature extraction of the data. Other alternatives have been proposed to replace the PCA with the independent component analysis (ICA) <cit.>, the kernel PCA (KPCA) <cit.> or other supervised methods such as the discriminant analysis feature extraction (DAFE) <cit.>, the non-parametric weighted feature extraction (NWFE) <cit.>, Sparse Hilbert Schmidt Independence Criterion and surrogate kernel (HSIC) <cit.>, etc. These methods can capture more spectral relations among hyperspectral bands and hence provide better spectral information than the PCA.

Recently, the vector strategy <cit.> has been investigated to effectively adapt APs on multispectral and hyperspectral image. The motivation of that work is to replace the marginal strategy, i.e. independently applying APs on each image band (or each component yielded by the aforementioned feature extraction methods) and stacking them to form the extended APs, with the vector strategy which can simultaneously process all available bands based on predefined vector-ordering relations. As a result, tree construction can be done once per multivariate image and the proposed vector APs (VAPs) become promising to deal with such hyperspectral data.

While the application of APs to optical remote sensing data has been strongly focused on, their exploitation to other remote sensing data is quite limited. One may witness some tentative work on polarimetric SAR images <cit.>, multispectral image derived features such as NDVI <cit.> or edge information <cit.> as well as on LiDAR data <cit.>. This is still an opened topic for on-going and future research in remote sensing imagery field.



 §.§ Tree formation


Tree formation is the first principal stage of the AP construction framework (Fig. <ref>). As described in Sec. <ref>, the standard APs <cit.> were computed based on one max-tree and one min-tree (i.e. both are component trees). Other work has been proposed to exploit the inclusion tree (i.e. tree of shapes) <cit.> in order to form the self-dual APs (SDAPs). The advantages of using such a tree of shapes are twofold. First, its self-dual property enables the attribute filtering operators to simultaneously access and model both dark and bright regions from the image. And secondly, by using only one tree of shapes to replace both min-tree and max-tree <cit.>, the feature dimension of SDAPs is reduced to half of that of APs. Consequently, SDAPs have been proved to be more efficient than APs in many research studies <cit.>. 

Since the above component and inclusion trees both rely on an ordering relation of the image pixels, their construction from multivariate images (e.g. multi- and hyperspectral data) is not straightforward. That is why the authors in <cit.> have recently investigated and proposed to use the partition trees such as α-tree and ω-tree to compute the α-APs, ω-APs, respectively. These profiles have been proved to provide fair performance compared to the standard APs. Moreover, they offer the possibility to work on multivariate images only using a single tree. Furthermore, it is also possible to rely on training samples to perform metric learning so as to provide the basic elements required for a partitioning tree <cit.>.



 §.§ Node attributes and threshold selection


The selection of tree node attributes as well as their thresholds for filtering on tree plays also an important role. Node attributes are usually related to the geometrical (such as size, shape) and statistical features (pixel distribution, texture, etc.) of the CC corresponding to the node. In the literature, four attributes have been used in most studies related to remote sensing image classification: area, standard deviation, moment of inertia, diagonal length of bounding box.

After deciding which attributes to calculate from nodes, the setting of their threshold values has been also concerned. Early work <cit.> usually set attribute thresholds manually based on experiments on some specific image data. However, since those values might be not applicable to other data, automatic threshold selection has drawn attention from many researchers. Some interesting studies have been proposed to automatically compute attribute thresholds using fixed formulas <cit.>, supervised approaches <cit.> as well as granulometric characteristic functions <cit.>. Readers are referred to the mentioned papers for further details about these attribute selection strategies.



 §.§ Tree filtering


Once the tree is formed and the attributes together with their thresholds are selected, the next stage is to evaluate each node in order to filter (i.e prune) the tree. Basically, there are two filtering rules including the pruning strategy (min, max, Viterbi decision rules) and the non-pruning strategy (direct, subtractive rules) <cit.>. Studies on the effect of different filtering rules have been done by <cit.>.



 §.§ Post-processing of output profiles


The output AP features, i.e. sequence of filtered images in Eq. (<ref>), can be directly fed into supervised classifiers such as SVM or Random Forest for classification task. Such direct application has provided better performance compared to MPs <cit.> in terms of classification accuracy as well as computational cost. However, since APs still involve quite redundant information within their high-dimension features, the post-processing of these profiles to improve their performance has been addressed in several studies. First and foremost, many studies have proposed to apply different feature selection techniques on APs to extract highly informative features and reduce their dimension. In <cit.>, both linear (PCA, ICA) and nonlinear methods (ICA, KPCA, DAFE, DBFE, NWFE, etc.) have been investigated. A general framework as well as a systematic survey on spatial-spectral approaches combining APs with these feature selection techniques have been investigated in <cit.>.

Other work has focused on extra spatial processing of APs for better characterization of structural and textural information from the image content. Recent studies believe that when dealing with VHR remote sensing images from which regions and objects become more heterogeneous, APs may not provide a complete spatial characterization of pixels. Therefore, some efforts have been proposed to replace each AP sample response by the histogram or some first-order statistical features of the local patch around that AP's pixel position. As a result, the local histogram-based APs (HAPs) <cit.> and the local feature-based APs (LFAPs) <cit.> have been proposed and proved to be more efficient for better dealing with local textures. Then, the extensions of these extra spatial processing methods on the self-dual profiles (using the tree of shapes) as well as on hyperspectral images have been provided <cit.>.

Last but not least, we refer readers to some other frameworks using the sparse representation <cit.> or the deep learning approach <cit.> for post-processing of AP features. Also, some ensemble methods <cit.> have been applied to better exploit and combine AP features to improve the classification performance.  




§ EXPERIMENTAL STUDY


This section describes our experimental study to evaluate the performance of the standard APs as well as some of their improvements and extensions. Supervised classification has been carried out on both panchromatic and hyperspectral image data in order to provide a comparative study. We first introduce the two data sets and the experimental setup. Then, classification results will be provided.



 §.§ Data description



  §.§.§ Reykjavik data set

The first data set is a panchromatic image of size 628 × 700 pixels acquired by the IKONOS Earth imaging satellite with 1-m resolution in Reykjavik, Iceland. This data consists of six thematic classes including residential, soil, shadow, commercial, highway and road. The image was provided with already-split training and test sets (22741 training samples and 98726 test samples). The input image together with its thematic ground truth map for testing and training sets are shown in Fig. <ref>(a).



  §.§.§ Pavia University data set
 
The second data set is the hyperspectral image acquired by the ROSIS airborne sensor with 1.3-m spatial resolution over the region of Pavia University, Italy. The image consists of 610 × 340 pixels with 103 spectral bands (from 0.43 to 0.86 μm) and covers nine thematic classes: trees, asphalt, bitumen, gravel, metal sheets, shadows, meadows, self-blocking bricks and bare soil. For this image, 3921 training samples and 42776 test samples were split for classification task. The false-color image (made by combining the bands 31, 56 and 102), the ground truth map and the training set are shown in Fig. <ref>(b). As previously discussed, for this data set, we first performed the PCA on the image and the first four PCs (involving more than 99% of the total variance) were preserved for our experiments.




 §.§ Setup

Supervised classification was conducted on the two data sets using the random forest classifier <cit.> with 100 trees. The number of variables involved in the training was set to the square root of the feature vector length. In order to evaluate and compare classification accuracy of different approaches, overall accuracy (OA), average accuracy (AA), and kappa coefficient (κ) have been taken into account. For attribute filtering, we exploited two attributes including the area and the moment of inertia. Ten area thresholds were adopted for the Reykjavik data as proposed by several papers <cit.>. For the Pavia University data, fourteen thresholds were automatically computed according to <cit.>. We have:

    λ_a,Rey =
    {25, 100, 500, 1000, 5000, 10 000,  
    
    	20 000, 50 000, 100 000, 150 000},


    λ_a,Pav =
    {770, 1538, 2307, 3076, 3846, 4615, 5384, 
    
    		 6153, 6923, 7692, 8461, 9230, 10000, 10769}.

Next, the manual settings used in many studies <cit.> were adopted for the moment of inertia attribute as follows:  

    λ_i,Rey = λ_i,Pav = {0.2, 0.3, 0.4, 0.5}.


In the following subsection, we report and compare the classification results yielded by the APs generated from different kinds of tree including: the max-tree (AP-maxT), the min-tree (AP-minT), one max-tree and one min-tree (standard APs) <cit.>, the SDAPs <cit.>, the α-APs and ω-APs <cit.>. We also provide the results of some effective post-processing techniques including the HAPs/HSDAPs <cit.>, LFAPs/LFSDAPs <cit.> and the deep learning approach (deep-APs) <cit.>. Then, for the hyperspectral Pavia data, VAPs <cit.> are evaluated as well. Here, we perform standard implementation as well as equivalent parameter configuration of these methods to ensure a fair comparison.



 §.§ Results

Tables <ref> and <ref> report the classification results of the Reykjavik and the Pavia data, respectively, yielded by the above mentioned methods. The calculation of each method's feature dimension can be consulted from the related papers. Here, we provide some remarks in terms of classification performance. For both data sets, we observe that those extension methods can provide extra classification accuracy compared to the standard APs but behave differently for each image. 


For Reykjavik image, efforts on changing the tree formation have provided some considerable effects. Indeed, the α-APs and ω-APs could outperform APs on each single max-tree or min-tree but still falls below the standard APs. Then, by using the tree of shapes, SDAPs significantly improved the accuracy with approximately 4% in OA (86.06% compared to 82.02%) and 5% in κ (0.824 compared to 0.773). Next, by post-processing the output profiles, techniques like HAP, LFAP and deep-AP have also provided important improvements. Consequently, the best classification result was obtained by using the local feature-based profiles  with =87.44% (κ=0.841) using min-tree and max-tree (LFAP) and =89.17% (κ=0.863) using the tree of shapes (LFSDAP). Compared to the standard APs, an OA enhancement of 5.42% and 7.15%, respectively, was achieved. 


For the hyperspectral Pavia data, we observe that APs built from different tree structures yielded different behaviors compared to the Reykjavik image. This time, the α-APs and ω-APs outperformed both APs and SDAPs. In particular, by using the ω-tree, one can achieve an =96.10%, i.e. 4.44% and 1.82% better than standard APs and SDAPs, respectively. For post-processing methods, VAPs and deep-APs provided better performance compared to HAPs and LFAPs. These methods have been proved to be efficient within a spatial-spectral context usually applied to hyperspectral data. As a result, the best classification accuracy was achieved by deep-APs with =99.02% and κ=0.979. Compared to the standard APs, an enhancement of 7.36% in OA and 9% in κ was adopted.



§ CONCLUSION


We have conducted a survey on recent developments from morphological attribute profiles in the context of remote sensing image classification. Three key components have been focused including the AP adaptation on multi-channel image data, the use of different tree representations and the various AP post-processing procedures. Experimental study on one panchromatic and one hyperspectral image has been performed to provide a general evaluation of different methods compared to the original framework. This paper may serve as an overview of AP recent advances to readers as well as a guidance to researchers working on this framework and its alternatives within their work. We believe the exploitation and adaptation of APs in remote sensing imagery still remains an open research topic for on-going as well as future work.



§ ACKNOWLEDGMENT

This work was supported the Région Bretagne grant and the BAGEP Award of the Science Academy and the Tubitak grant 115E857. 
The authors would like to thank Prof. Jon Atli Benediktsson and Prof. Paolo Gamba for making available the Reykjavik image and the hyperspectral Pavia University data.



IEEEtran



