A Multi-objective Optimization Benchmark Test Suite for Real-time Semantic Segmentation
Yifan Zhao Zhenyu Liang Zhichao Lu Ran Cheng














0009-0000-8943-9500

  Southern University of Science and Technology
  Shenzhen
  Guangdong
  China

AlpAcA0072@gmail.com




0009-0003-8876-8638

  Southern University of Science and Technology
  Shenzhen
  Guangdong
  China

zhenyuliang97@gmail.com





0000-0002-4618-3573

  City University of Hong Kong
  Hong Kong
  China
  43017-6221

luzhichaocn@gmail.com




0000-0001-9410-8263
Corresponding Author

  Southern University of Science and Technology
  Shenzhen
  Guangdong
  China

ranchengcn@gmail.com
















As one of the emerging challenges in Automated Machine Learning, the Hardware-aware Neural Architecture Search (HW-NAS) tasks can be treated as black-box multi-objective optimization problems (MOPs). 
An important application of HW-NAS is real-time semantic segmentation, which plays a pivotal role in autonomous driving scenarios. 
The HW-NAS for real-time semantic segmentation inherently needs to balance multiple optimization objectives, including model accuracy, inference speed, and hardware-specific considerations. 
Despite its importance, benchmarks have yet to be developed to frame such a challenging task as multi-objective optimization.
To bridge the gap, we introduce a tailored streamline to transform the task of HW-NAS for real-time semantic segmentation into standard MOPs. 
Building upon the streamline, we present a benchmark test suite, CitySeg/MOP, comprising fifteen MOPs derived from the Cityscapes dataset. 
The CitySeg/MOP test suite is integrated into the EvoXBench platform to provide seamless interfaces with various programming languages (e.g., Python and MATLAB) for instant fitness evaluations.
We comprehensively assessed the CitySeg/MOP test suite on various multi-objective evolutionary algorithms, showcasing its versatility and practicality. 

Source codes are available at <https://github.com/EMI-Group/evoxbench>.









<ccs2012>
   <concept>
       <concept_id>10003752.10003809.10003716.10011136.10011797.10011799</concept_id>
       <concept_desc>Theory of computation Evolutionary algorithms</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10010405.10010481.10010484.10011817</concept_id>
       <concept_desc>Applied computing Multi-criterion optimization and decision-making</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Theory of computation Evolutionary algorithms
[500]Applied computing Multi-criterion optimization and decision-making


























A Multi-objective Optimization Benchmark Test Suite for Real-time Semantic Segmentation
    Ran Cheng
    April 29, 2024
=======================================================================================






§ INTRODUCTION

Neural Architecture Search (NAS), a pivotal component of Automated Machine Learning <cit.>, has traditionally focused on single-objective optimization problems, aiming to minimize prediction errors. This paradigm involves three key processes: search space design, search strategy formulation, and architecture performance evaluation.

Recent advancements have introduced NAS with multi-objective optimization, particularly in Hardware-aware Neural Architecture Search (HW-NAS). 
HW-NAS automates the search for deep neural network architectures, aligning them with specific applications <cit.>, which is crucial for resource-constrained platforms. 

Real-time semantic segmentation emerges as a key challenge in this domain, meeting the need for high computational efficiency with accurate real-time processing.

Semantic segmentation, a cornerstone of computer vision, assigns semantic labels to each image pixel. However, the shift towards real-time applications, such as autonomous driving, has underscored the necessity of considering multiple objectives in model design, including not only hardware-unrelated metrics such as model accuracy (measured as mean Intersection over Union, mIoU), the scale of models but also hardware-related inference speed.
This emerging demand highlights the significance of HW-NAS in devising architectures that successfully balance these diverse optimization objectives.

Due to the multiple objectives in designing and optimizing the DNN architectures, the HW-NAS tasks can be fundamentally viewed as a Multi-objective Optimization Problem (MOP).
Meanwhile, since the black-box optimization characteristics of the HW-NAS, conventional optimization methods encounter significant challenges in addressing such problems. 

Therefore, it is intuitive to adopt representative Multi-Objective Evolutionary Algorithms (MOEAs) <cit.> for HW-NAS. 
However, the application of MOEAs within the HW-NAS field is relatively under-explored. 
Existing NAS work on semantic segmentation<cit.> lacks capabilities of hardware awareness or constraints on real-time.

To address the challenges, this paper proposes a multi-objective optimization benchmark test suite for real-time semantic segmentation, dubbed CitySeg/MOP. The benchmark introduces an architecture search space for real-time semantic segmentation and tailors two categories of performance evaluators covering the multiple objectives. 
The benchmark test suite is seamlessly integrated into the platform EvoXBench <cit.> and provides interfaces with various programming languages (e.g., Python and MATLAB) without the requirement for any machine learning accelerating libraries. This fills the gap left by the lack of a multi-objective optimization benchmark test suite for real-time semantic segmentation in the HW-NAS.



§ METHOD






 §.§ Problem Formulation

The HW-NAS problem can be formulated as a MOP, which can be defined as follows <cit.>:

    min_x    F(x) = (f^e(x;ω^*(x)), f^c(x), f^ℋ(x))
    s.t.    ω^*(x)=ωargmin.ℒ(x;ω),  x∈Ω_x.
      ,

where

    f^c(x) = [ f_1(x)      ⋯ f_m(x) ],

and 

    f^ℋ(x)    = [   f_1^h_1(x)            ⋯   f_n^h_1(x);            ⋮            ⋱            ⋮; f_1^h_|ℋ|(x)            ⋯ f_n^h_|ℋ|(x);              ],
    ℋ    = {h_1,…,h_|ℋ|},

where Ω_x is the architecture search space, x is an encoded network architecture, ω(x) is the weights vector of the network architecture x. 
For a dataset 𝒟, there is 𝒟={𝒟_train,𝒟_valid,𝒟_test}. ω^*(x) is the optimal weights vector of the network architecture x that minimizes loss ℒ on 𝒟_valid. 
F(x) is the objective vector, which consists of three categories: prediction error (f^e), complexity-related objectives (f^c), and hardware-related objectives (f^ℋ), in which ℋ denotes the set of hardware platforms. 

In f_n^h_|ℋ|(x), the h_|ℋ| denotes the |ℋ|-th hardware platform and n denotes the n-th objective.









 §.§ Search Space


The search space adopted is inspired by MoSegNAS <cit.>. The search space's overall structure comprises one stem layer, four stage layers, and one tail layer. The stem and stage layers contain cells that need to be searched, while the tail layer remains constant to be a global average pooling layer.
Each cell comprises a sequence of convolutions, which is a typical residual bottleneck block.

In addition, the search space encodes essential attributes, including depth, expansion ratio, and width (denoting the channels). 
The specific form of the coded architecture is shown in Figure <ref>.



 §.§ Fitness Evaluator

We use different fitness evaluators to cope with the diversity of the objectives. There are two categories of fitness evaluators and three categories of objectives, where the surrogate model is used for f^e, and the look-up table is used for f^c and f^ℋ.







The fitness evaluator of inference accuracy objective (f^e) is measured in (1 - mIoU). The mIoU is defined as follows:

    mIoU=1/N∑_n=1^NIoU=1/N∑_n=1^NTP/(TP+FP+FN),

where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels of the segmentation result in the given image, and N is the number of the defined classes over the whole test set. 


It is computationally unacceptable to train from scratch to assess the prediction accuracy. Rather than adopting the proxy methods (with fewer training epochs or gradient approximation<cit.>), we adopted the surrogate model to approximate the mapping from the network architectures' space to the inference accuracy objective (f^e). 
In this paper, we adopt a Multi-Layer Perceptron (MLP) with ranking loss and mean square error (MSE) loss as the surrogate model:

    1ℒ    = ℒ_rank + ℒ_mse,
    ℒ_mse    = 1/N∑_iŷ_i-y_i^2,
    ℒ_rank    = 1/2N∑_i,jmax(0,γ-δ(ŷ_i,ŷ_j)(y_i-y_j)), 
    δ(ŷ_i,ŷ_j)    = 1,if ŷ_i>ŷ_j,
    -1,otherwise,

where the ŷ_i is the output of the MLP when given input x_i. 
The γ is the margin controller hyperparameter set to 5 × 10^-2. 





We have the assumption that complexity-related objectives (f^c) and the hardware-related objectives (f^ℋ) are decomposable, i.e. the objectives can be decomposed to the sum up of the layers' corresponding objectives. Under this assumption, we can enumerate all the values of the stems and the stages individually to gain the look-up table. Thus, we can compose the architectures' stem and stages by summing up the values of f^c and f^ℋ to gain the overall result.
The procedure of the whole fitness evaluation is shown in Figure <ref>. 
The f^ℋ can be affected by stochastic perturbation with the coefficient β to simulate the hardware's performance fluctuation. The β is set to [0.95, 1.05] for f^ℋ_1 and [0.98, 1.02] for f^ℋ_2 in this paper.


Unlike the stochastic gradient descent process, the approximated optimal weights of the candidate network architecture are inherited from the trained hypernetwork. 
Hypernetwork essentially introduces a mapping from the network architecture space ℤ^d to the weight space ℝ^m, i.e. Ψ: ℤ^d →ℝ^m. Ψ can be formulated as follows:

    Ψ(x) = ω argmin ℒ(x;ω),

where d is the dimension of network architecture space ℤ and m is the dimension of weights space ℝ.






 §.§ Benchmark Generation

The network architectures are trained and tested on the Cityscapes dataset <cit.>, which is a large-scale dataset for urban city street semantic segmentation. 
Input images with a low expand ratio can significantly reduce memory usage and computation resource overhead.
Thus, we provide images at different expansion ratios in the search space.






Based on EvoXBench <cit.>, we construct a tailored benchmark test suite CitySeg/MOP for real-time semantic segmentation. As summarized in Table <ref>, there are fifteen instances in total, D indicates the number (dimension) of decision variables, and M indicates the number of objectives in the problems. 
For deployment, there are two hardware devices involved: the NVIDIA GeForce RTX 4090 (h_1) and Huawei Atlas 200 DK (h_2), where the former is a representative GPU device and the latter is a typical ARM-based edge computing device with only 9.5W energy consumption. 
The CitySeg/MOP test suite's problems are divided into three categories. The first five problems contain h_1's objectives, and the sixth to the tenth problems contain h_2's objectives. The last five problems enable the hardware-aware features by combining h_1's and h_2's objectives.
Table <ref> in Appendix <ref> provides the definition of objectives in CitySeg/MOP test suite.

As summarized in Table <ref>, the test instances are listed for hardware and number of objectives in ascending order, from two to seven objectives. 



§ EXPERIMENTS

This section provides the experimental results for comprehensively assessing the proposed benchmark test suite CitySeg/MOP. 
Firstly, we conducted experiments to assess the prediction accuracy, sample diversity, and evaluation efficiency of the predictors; afterward, we conducted benchmark tests running on the proposed test suite.
The results of evaluation efficiency and benchmark tests are obtained by running each algorithm 31 times on EvoX <cit.> or PlatEMO <cit.> independently, with 10,000 fitness evaluations. 





 §.§ Prediction Accuracy


The correlations between the measured and the predicted metrics are shown in Figure <ref>. For Figure <ref> and Figure <ref>, we measured the number of floating point operations (FLOPs) (f^c_1) and the number of parameters (f^c_2) of the smallest unit. Network architectures' FLOPs (f^c_1) and parameters (f^c_2) are weighed against the corresponding parameters of the smallest units. The Mean Absolute Error (MAE) of the FLOPs and the parameters is 0.0. For Figure <ref>, the MAE and ρ of the prediction error (f^e) are 1.71× 10^-2 and 0.9960 respectively. Figure <ref> shows the accuracy of the predictions for f^e, f^c_1 and f^c_2.



 §.§ Sample Diversity




Figure <ref> shows the randomly sampled architectures' distribution of FLOPs (f^c_1) and number of parameters (f^c_2). 
The FLOPs (f^c_1) has multiple peaks, which is normally distributed, and the values within each peak are also normally distributed. 
The short tail on the left occurs because network architectures with the depth of all stages being zero are not permitted. Conversely, the long-tailed peaks on the right represent scenarios where no stage has zero depth.

Figure <ref> in Appendix <ref> shows the randomly sampled architectures' correlation between hardware-related consumption and FLOPs, the number of parameters. We can see from Figure <ref> that data is distributed in multiple clusters, resulting from the stages in the search space. This is consistent with the data distribution in Figure <ref>.




 §.§ Evaluation Efficiency

Table <ref> in Appendix <ref> provides the evaluation efficiency when called by Python or MATLAB's interface. 
The evaluation is performed on a local machine with one CPU core using PlatEMO <cit.> for MATLAB, and NVIDIA GeForce RTX 4090 using EvoX <cit.> for Python respectively.





 §.§ Benchmark Tests


As a demonstration, we conducted benchmark tests on the proposed CitySeg/MOP test suite using six representative MOEAs falling into three different categories: (1)  NSGA-II <cit.> and NSGA-III <cit.> (dominance-based); (2) MOEA/D <cit.> and RVEA[We use the version with the reference vector regeneration strategy, which is also known as the RVEA*.] <cit.> (decomposition based); (3) IBEA <cit.> and HypE <cit.> (indicator based).

Table <ref> in Appendix <ref> summarizes the statistical results of the HV values of the test instances on the CitySeg/MOP test suite. 
Of these test instances, in terms of solving test instances with MOPs and MaOPs (Problems with more than three objectives), algorithms NSGA-II, NSGA-III (dominance-based methods) have an advantage. IBEA, HypE (indicator based methods) perform better on certain problems.

Figure <ref> in Appendix <ref> shows the non-dominated solutions obtained by algorithms.

By performing MOEAs on CitySeg/MOP15, algorithms favor low energy-consumption, low-latency network architectures. However, these algorithms are inconsistent in the choice of prediction error objective and model scale objectives.



§ CONCLUSION


In this paper, we introduce CitySeg/MOP, a multi-objective optimization benchmark test suite tailored for real-time semantic segmentation. 
By considering HW-NAS tasks real-time semantic segmentation as standard MOPs, CitySeg/MOP encompasses various objectives such as accuracy, computational efficiency, and real-time constraints. 
The experimental results demonstrate the efficiency, accuracy and diversity of CitySeg/MOP. 
Moreover, by providing seamless integration to the EvoXBench <cit.> platform, it enables efficient interfaces to general MOEAs implemented in various programming languages.
As a standard multi-objective optimization benchmark test suite tailored for real-time semantic segmentation, CitySeg/MOP contributes to filling the gap between training EMO research and its applications in complex HW-NAS tasks.















ACM-Reference-Format









§ EXPERIMENTAL SETUP


The experiments were conducted on EvoX <cit.> for Python, and PlatEMO <cit.> for MATLAB.














§ SUPPLEMENTARY RESULTS
















