Re-identification = Retrieval + Verification: Back to Essence and Forward with a New Metric

Re-identification (re-ID) is currently investigated as a closed-world image retrieval task, and evaluated by retrieval based metrics. The algorithms return ranking lists to users, but cannot tell which images are the true target. In essence, current re-ID overemphasizes the importance of retrieval but underemphasizes that of verification, \textit{i.e.}, all returned images are considered as the target. On the other hand, re-ID should also include the scenario that the query identity does not appear in the gallery. To this end, we go back to the essence of re-ID, \textit{i.e.}, a combination of retrieval and verification in an open-set setting, and put forward a new metric, namely, Genuine Open-set re-ID Metric (GOM). GOM explicitly balances the effect of performing retrieval and verification into a single unified metric. It can also be decomposed into a family of sub-metrics, enabling a clear analysis of re-ID performance. We evaluate the effectiveness of GOM on the re-ID benchmarks, showing its ability to capture important aspects of re-ID performance that have not been taken into account by established metrics so far. Furthermore, we show GOM scores excellent in aligning with human visual evaluation of re-ID performance. Related codes are available at https://github.com/YuanXinCherry/Person-reID-Evaluation


Introduction
Re-identification (re-ID) is the task of finding out the presence of person/vehicle-of-interest in a database of images captured from multiple different cameras in a widearea cross-camera video surveillance scenario [24]. The re-ID task is one of the growing communities of computer vision research, and is imperative for the intelligent video surveillance system. Yet, the evaluation of almost all re-ID algorithms is done under a closed-world retrieval set-* Equal contributions. Zheng Wang is the corresponding author.  1: Illustration of existing retrieval and verification procedures. 1) Closed-world retrieval: the method returns a ranking list to users and cannot tell which images are the true targets. The user needs to judge targets according to their experience and feeling. 2) Closed-world verification: given a decision boundary, images whose distances are below the boundary are considered as the targets. Nevertheless, the method cannot distinguish ground truth (GT) and non-GT within the boundary. 3) Open-set: the GT of the probe does not always exist in the gallery, thus re-ID procedure should include this kind of scenario.
ting. Under this setting, existing evaluation metrics can be classified into two types according to the number of ground truth (GT) that exists for each query. The first type is a single-GT evaluation with Cumulative Matching Characteristics (CMC) [7]. The second type is a multi-GT evaluation with mean Average Precision (mAP) [27]. Also, a novel metric named mean inverse negative penalty (mINP) [25], as a supplement, has been proposed to evaluate the property of the hardest correct match. However, existing retrieval based metrics are not able to reveal which algorithms are outstanding in the scenario that the system needs to output some targets automatically rather than a ranking list with no clear targets. Users still need to find out GTs from returned ranking lists via their eyes (retrieval in Figure 1). In essence, the procedure of re-ID expects machines to out-  [7] closed-world Retrieval mAP [27] / mINP [25] closed-world Retrieval precision [14] / recall [12] closed-world Verification FAR [15] / FTR [29] open-set Verification GOM open-set re-ID put targets automatically without human involving, so that further tracking, behaviour analysis, and investigation will continue smoothly. To our knowledge, few methods [12,14,29] studied and formulated re-ID as a verification task. Similar to the detection and classification tasks, where, given a threshold, the algorithms can be evaluated by the metrics of precision [14] and recall [12] on the automatically predicted targets. However, the verification based metrics are not able to indicate which returned targets are GTs and which GTs are not returned (verification Figure 1). In practice, it is still required that the true positive ranks top in predicted targets. Thus, we consider that re-ID cannot be formulated as retrieval or verification alone. It is essentially a combination of retrieval and verification. In this paper, we propose to design new metrics to benefit the re-ID community with a comprehensive evaluation and making it forward in its right way.
In addition, current re-ID methods have made an unscalable closed-world assumption, where every probe identity must guarantee to appear in the gallery set. Nevertheless, the GT of the probe does not always exist in the gallery in real-world applications. Typically, it is frequent to search for an inconclusive suspect who may not be in a video or a large set of videos in realistic forensic video analysis. In this case, it is called an open-set problem (open-set Figure 1), which has been less investigated [15,29,35]. Specifically, the false accept rate (FAR) [15] and the false target rate (FTR) [29] were designed to tell the proportion of nontarget probes verified as the target. They are also often reported with verification previsions by the Receiver Operating Characteristic (ROC) [15] under different decision thresholds. However, open-set based metrics above mainly focus on the probes that do not have GTs in the gallery, not comprehensively evaluating the performance of re-ID algorithms.
To summarize the disparities between our novel metrics and existing metrics, a comparison is illustrated in Table 1. To remedy the above problems of existing metrics (i.e., not really re-ID procedure), we present a novel Genuine Openset re-ID Metric (GOM) in this paper, including mean re-ID Precision curve (mReP ) and mean False Rate curve (mF R) respectively. mReP and mF R together are capable of evaluating genuine re-ID in both closed-world and open-set, promoting machines to automatically select and return target images. We will provide analysis as to why GOM is preferable to current alternatives for evaluating re-ID algorithms. In essence, current re-ID overemphasizes the importance of retrieval but underemphasizes that of verification as well as the open-set problem. GOM explicitly measures both properties of the two tasks, and combines these in a balanced way.
The GOM metric is also intuitive to understand (please refer to Table 2 and its illustration in Section 6). It consists of several sub-metrics. The Retrieval Precision RP is similar to AP. The Verification Precision V P is simply the percentage of returned GTs between GTs and returned negatives. The ReP score is the geometric mean of RP and V P over different decision thresholds. The false rate F R is simply the percentage of false targets. The final two metrics are mReP , averaged over all ReP , and mF R, averaged over all F R. We make three significant novel contributions: 1) We propose a novel metric GOM for evaluating genuine open-set re-ID. GOM can be decomposed into a family of sub-metrics which are able to evaluate different aspects of re-ID separately. This enables a clear understanding of the different types of tasks that the feedback rank lists are returned based on threshold, and enables decision thresholds to be tuned for different requirements.
3) We evaluate GOM on two person re-ID datasets (Market-1501 [27] and DukeMTMC-reID [31]) and one vehicle re-ID dataset (VeRi776 [16]), and analyze its properties compared to other metrics for evaluating current stateof-the-art re-ID methods. Besides, we have exhibited a user study comparing how different metrics align with the human judgment of re-ID and show that GOM well aligns with the desired re-ID's procedure of users.

Related Work
History of re-ID metrics. The re-ID task originates from multi-camera tracking [23]. In the early years, re-ID datasets were collected by two cameras, such as VIPeR [8], and each query had only one GT. Hence, early works tended to evaluate using the Cumulative Matching Characteristic (CMC) [7] curve. It has often been highly criticized [27] for its properties that focus on the most similar GT. Recently, large scale datasets, such as Market-1501 [27] and DukeMTMC-reID [31], have been constructed, which contain multiple GTs. To improve the limitations of CMC, Zheng et al. [27] proposed to use the mean average precision (mAP) for evaluation, which is originally widely used in image retrieval. It can address the issue of two sys-tems performing equally well in searching the first GT, but having different retrieval abilities for other hard GTs. Recently, mINP [25] was proposed to evaluate the ability to retrieve the hardest GT, which avoids the domination of easy matches in the mAP/CMC evaluation. On the other hand, the precision and recall metrics, specifically proposed for detection and classification tasks, have been used by few re-ID methods [12,14], due to its focus on measuring the verification accuracy rather than the retrieval accuracy. However, they exhibit unintuitive and atypical in regards to retrieval. Metrics of open-set re-ID. Closed-world re-ID aims to determine which gallery image is the probe. However, numerous irrelevant identities exist, and the probe identity may not appear in the gallery, i.e., open-set re-ID. Open-set re-ID is usually formulated as a verification problem [15,29,35]. Zheng et al. [28] conducted the first open-set re-ID work. Liao et al. [15] decomposed open-set re-ID into detection and identification, where identification rate (DIR) and false accept rate (FAR) were discussed. Zheng et al. [29] utilized the true target rate (TTR/FTR similar to DIR/FAR) for performance evaluation. Note that the Receiver Operating Characteristic (ROC) [15] is made up of DIR and FAR. Although the open-set re-ID is much closer to practical video surveillance applications than the closed-world, the attention devoted to this issue is relatively limited for three possible reasons. First, closed-world re-ID is a utilizable technical route, and it is convenient and fair for conducting research on previous theories and technologies due to various baselines and datasets. Second, the low recognition rates under low false accepted rates of existing methods show that open-set re-ID is challenging. Third, there are no public datasets designed for open-set re-ID evaluation. Multi-Target Multi-Camera Tracking (MTMCT). MTMCT aims to determine the cross-camera trajectories of certain targets captured from multiple cameras [2]. MTMCT and open-set re-ID could fall under the crosscamera identity verification task.
Recently, Ristami et al. [18] presented a CNN feature for both MTMCT and re-ID. However, MTMCT still has several differences from re-ID. The former focuses on reducing classification error rates, whereas the latter pays attention to improving the ranking performance at a certain error-tolerant rate. Meta-Evaluation of metrics. Vasant et al. [22] investigated the trade-off between providing multiple evaluation metrics versus a single unifying metric. Compared to multiple metrics which are helpful for researchers or developers to debug algorithm, a unified metric can be useful to assist end-users to easily choose effective models from various options. Existing re-ID metrics have different views, but are designed for the retrieval task or verification task, essentially, not suitable as a unified metric for genuine re-ID. Srikrishna et al. [21] also pointed out that researchers should take a broader view of how algorithms perform and not just look at raw rank-1 or mAP numbers. We present GOM as a suitable solution for unified comparison.

Preliminaries
This section describes the re-ID task, the role of evaluation metrics, as well as the definition of open-set re-ID.
What is re-ID? Re-ID is one of the core tasks for the intelligent video surveillance system. The input is a query/probe that came from the human need. The output is a batch rank list of gallery based on distance/similarity that contains the information about: 1) whether this target exists or appears in the gallery (open-set)? 2) whether all targets are found out in the ranking list (verification)? 3) which images belong to the same identity (retrieval). Evaluation metrics. The choice of evaluation metric is extremely important, as the properties of the metric determine how different errors contribute to a final score. The choice of metric also has the ability to heavily influence the direction of re-ID research. In the age of competitive benchmarks, a lot of researches are evaluated on the ranking ability on the benchmarks. This will also guide researches and methods towards focusing on these aspects. Definition of open-set re-ID. Suppose that there are two query sets Q + and Q − . Q + consists of N images that have the GTs in the gallery set G, and Q − consists of M images that do not have the GTs in G.

Overview of Previous Metrics
A brief revisit of the current commonly used evaluation metrics is as follows:

CMC
For each query i ∈ Q + , all gallery images k ∈ G are ranked based on the distance d(i, k). Cumulative Match Characteristic (CMC) [7] counts if top-x ranked gallery samples contain the query identity. The rank index of the correct match is denoted as R k * i . The gallery image k * has the same identity as the query i. CM C@x is defined as: where I(·) is the measure function. Note that CMC does not have a common agreement when it comes to the multigallery-shot setting, where each gallery identity could have multiple instances.

mAP
For each query i ∈ Q + , an average precision (AP) [27] calculates the area under the Precision-Recall where G i stands for the sub gallery set that have the same person ID as the query i. Then, mAP evaluates the overall performance:

mINP
The negative penalty (NP) [25] measures the penalty to find the hardest correct match indicates the rank position of the hardest match, and |G i | represents the total number of correct matches for query i. For consistency with CMC and mAP, the inverse negative penalty (INP) is preferred. Overall, the mean INP (mINP) of all the queries is represented by:

FAR and ROC
Liao et al. [15] considers the open-set re-ID problem. It defines an identification rate (DIR) and a false accept rate (FAR) where τ is the decision threshold, x is the rank index. Receiver Operating Characteristic (ROC) curve considers a situation that the query is not present in the gallery. Given the rank x and the decision threshold τ , an ROC curve can be drawn by plotting DIR versus FAR.

The GOM Evaluation Metric
The main contribution of this paper is a novel evaluation metric for evaluating re-ID performance. We term this evaluation metric GOM (Genuine Open-set re-ID Metric). GOM builds upon some previously used metrics, while addressing many of their deficits. GOM is designed to: 1) provide scores for re-ID evaluation which combines different aspects of re-ID evaluation, 2) evaluate all over stage performance under different decision threshold, and finally, 3) decompose into sub-metrics which allow an analysis of the different components of retriever's or verifier's performance. Retrieval Precision. For a particular decision threshold τ , we use AP to record the retrieval precision of each query i: Verification Precision. Also, given a decision threshold τ and a query i, the verification precision can be calculated as: where |G T P τ,i |, |G F N τ,i |, and |G F P τ,i | represent the number of correct match subset (true positive), the number of incorrect match subset (false positive), and the number of targets not be found (false negative) in the returned subset under the decision threshold τ . Re-ID Precision. Then, the genuine re-ID precision mReP evaluates the mean of the geometric mean of the retrieval precision and the verification precision by Eq. 6. This formulation ensures that both retrieval and verification are balanced. Note that it evaluates the overall performance of all queries that consist of targets in the gallery.
False Rate. We evaluate the false rate for all queries Q − with no GTs in the gallery. For a query j, F R τ,j calculates the proportion of incorrect matches: where B is an integer constant 1 to decide the false rate. Then, mF R is used to evaluates the overall performance of all queries that do not consist of GTs in the gallery, as Eq. 8 represents.
Integrating over decision thresholds. According to different τ , we can draw an mReP curve and an mF R curve. The re-ID precision and false rate scores are the integral of mReP and mF R scores across the valid range of τ values between 0 and 1. The results are calculated via the following formulation 2 : The evaluation process has three steps: 1) Given a distance matrix (|Q| × |G|) generated by the evaluated method, we normalize it to [0, 1] and set a series of thresholds. 2) We calculate the mReP τ and mF R τ at such given threshold τ by Eq. 6 and Eq. 8. Then, we draw the mReP and mF R curves.
3) The M REP and M F R under different thresholds are calculated, and the important thresholds τ max for mReP τ and τ nz for mF R τ are recorded as well. Figure 2: A family of metrics of GOM. Note that mReP τ , mRP τ , mV P τ and mF R τ stand for the mean value of ReP τ , RP τ , V P τ and F R τ scores by all queries, respectively. ReP τ is calculated by the geometric mean of RP τ and V P τ . mReP max and mV P max stand for the max value of mReP τ and mV P τ under all thresholds τ .

Analysis of GOM
In this section, we make an analysis of GOM in terms of taxonomy of error types, and use an example to show the advantage of our metrics. Analysis of re-ID errors. We classify potential re-ID errors into three categories: retrieval errors, verification errors, and decision errors. 1) Retrieval errors occur when negative samples rank higher than positive samples in the gallery. We always use the metrics of CMC [7], mAP [27], and mINP [25] to evaluate different methods. These retrieval based metrics all focus on the rank of samples in the whole gallery while ignoring which the real targets are. 2) Verification errors occur when the negative samples are in the returned results, or when the positive samples are not in the returned results. We often use the metrics of precision and recall to evaluate different methods. Similar to detection/classification evaluation, verification based metrics only focus on the proportion of returned/missed positives, but cannot tell the exact correct matches. 3) Decision errors occur on the condition that methods obtain returned samples when the query identity does not exist in the gallery set. FAR/FTR is the most representative metrics in the open-set setting. These evaluation metrics only reflect whether the query identity appears in the gallery, but are impossible to measure the correct match of returned results.
To better assess these three types of errors, we propose the novel evaluation metric GOM (see in Sec. 5). Figure 2 shows the set of sub-metrics of GOM and their relations. The sub-metrics have two main purposes. The first purpose is to enable a simple comparison between methods to determine which performs better than the others. The second purpose of evaluation metrics is to enable the analysis of different types of errors, as well as the understanding of the different types of requirements. For the retrieval errors, we improve original average precision AP by adding a thresh-old τ and design the RP τ (Eq. 4) to compute retrieval accuracy in terms of thresholds. For the verification errors, to balance existing precision and recall, we propose the verification precision V P τ (Eq. 5), similar to the standard Jaccard index that is commonly used for evaluating classification. Note that V P τ can be represented by precision and recall. Please refer to the supplementary for its detail. We can also consider both retrieval and verification errors, then combine RP τ and V P τ together using geometric mean ReP τ (Eq. 6) under the same threshold. For the decision errors, we also propose a false rate F R τ (Eq. 8) in terms of the threshold, which shows whether fewer negatives are returned, while the existing metric FAR/FTR does not have this kind of ability. Why geometric mean for ReP. The ReP τ formulation contains a square root operation after the multiplication of V P τ and RP τ . This square root has three effects. 1) The scores of the model can be increased in terms of the magnitude and spread. It is nice to observe that both the magnitude and spread of ReP τ scores fall in the same range as previous metrics. This indicates that researchers' current intuitive understanding of how good certain scores are still roughly holds. 2) ReP τ can evenly balance both retrieval and verification via calculating the geometric mean of these two scores. The geometric mean is more suitable in re-ID than other formulations such as the arithmetic mean. When a model completely fails in either retrieval or verification, the score of the geometric mean can approach 0 to effectively represent this. 3) Double counting of similar error types can be taken into consideration. As discussed above, retrieval/verification errors are included in the ReP τ score. These errors can be prevented via the use of the square root. Comparison with existing metrics using toy samples. Table 2 shows some toy samples and their results under existing metrics CMC@1, AP, INP, and FAR, and our metrics RP τ , V P τ , ReP τ , and F R τ . The examples demonstrate the following conclusions: • For the rank list I and II, the figure shows that their retrieval results are the same. CMC, AP, and INP metrics also give the same scores. Actually, the verification results are different. For example, under threshold τ 1 , list I outputs two targets, while list II does not output any targets. This example shows that the results may have the same retrieval score, but different verification scores. In this case, existing metrics designed for evaluating retrieval (CMC, AP, and INP) are ineffective.
Our metric V P τ shows the verification performance under different thresholds. V P τ clearly reflects that the results of the rank list I (0.67 at τ 1 and 1 at τ 2 ) are better than that of rank list II (0 at τ 1 and 0.33 at τ 2 ). Furthermore, in order to comprehensively consider the retrieval and verification, we proposed evaluation metric ReP τ by making a geometric mean of AP τ and  Table 2: A simple re-ID example highlighting the main differences between evaluation metrics. Six different ranking lists are shown. AP and F AR overemphasize the task of retrieval and verification respectively. GOM balances both of these by being an explicit combination of a retrieval precision RP and a verification precision V P , contributed to re-ID precision ReP , and assess the false rate under the different threshold by F R.
V P τ , it also demonstrates that the rank list I is preferable, even though existing metrics cannot distinguish them.
• For the rank list III and IV, the figure shows that their verification results are the same under thresholds τ 1 (both return one target) and τ 2 (both return three positives and one negative). Existing metrics (CMC, AP, and INP) cannot reflect this situation, but our metric V P τ demonstrates that their verification scores are the same (0.33 at τ 1 and 0.75 at τ 2 ). These two lists are indeed different when evaluating with retrieval metrics. Our metric RP τ at τ 2 shows that the result of rank list IV (0.92) is better than that of rank list III (0.81). This example shows that the results may have the same verification score, but different retrieval scores. Furthermore, the metric ReP τ also demonstrates that the rank list IV is preferable, even though existing metrics and the verification metric cannot distinguish them.
• For the rank list V and VI, they demonstrate the situation of open-set re-ID. The figure shows that the rank list VI performs relatively better than the rank list V, because more negatives are in a larger distance. However, if we use existing metrics FAR τ /FTR to measure the results. The rank list V and VI get the same score (0 at τ 1 and 0 at τ 2 ). On the other hand, our metric F R τ at τ 2 clearly reflects that the result of rank list VI (0.2) is better than that of rank list V (0.4).
To sum up, currently used metrics overemphasize the importance of retrieval but underemphasize that of verification, while our set of metrics can perform comprehensive with retrieval, verification, and open-set tasks. GOM decomposes into a family of sub-metrics that are able to evaluate different aspects of re-ID separately, and it also enables a clear understanding of different aspects of re-ID under different thresholds.

Experimental Validation
Sec. 6 provides a theoretical analysis of the re-ID errors, and compares GOM with existing metrics via toy samples. In this section, we verify the effectiveness of GOM in the actual application task. First, we conduct qualitative and quantitative experimental evaluations on person and vehicle re-ID tasks. Second, we verify the effectiveness and evaluation quality from the level of human visual assessment study.
To simulate the open-set re-ID setting, we randomly select 100 images from other datasets as new queries. In person re-ID evaluations, we compare a batch of state-of-the-art methods, both supervised and unsupervised. Especially, we discuss the performance with GOM under different thresholds and give some explorative solutions. We restrict our evaluation to only those methods that are published in peerreviewed journals and conferences. We evaluate 12 supervised methods [1, 3, 9-11, 13, 17, 19, 25, 26, 30, 33] and 5 unsupervised methods [4-6, 20, 32] on Market-1501 and DukeMTMC-reID, and 3 vehicle re-ID methods [9,17,34] on VeRi776. Datasets. Market-1501 [27] is a large scale person re-ID dataset, containing 32,688 labeled images, 12,936 images of 751 identities for training, and 19,732 images of 750 identities for testing. DukeMTMC-reID [31] is also a popular evaluated person re-ID dataset, which has 16,522 training images, 2,228 query images, and 17,661 gallery images. VeRi776 [16] is a typical vehicle re-ID dataset, which consists of over 50,000 images of 776 vehicles. Re-ID results by GOM. Figure 3 shows the mReP and mF R curves of all evaluated methods on Market-1501 and DukeMTMC-reID datasets. In addition to the results by existing metrics CMC@1 and mAP, Table 3 shows the results of mV P max , mReP max , M REP , and M F R on two per-   son re-ID datasets. Table 4 shows the results on the vehicle re-ID dataset. The detailed analysis is as follows: 1) Figure 3 shows that the curves of all methods have highly consistent trends and shapes on two different datasets. The pros and cons of different methods on two datasets are also extremely similar. These demonstrate the good universality of our metric.
2) Table 3 shows that the method has excellent performance evaluated by retrieval based metrics (mAP), whereas it may not achieve good performance evaluated by verification based metrics (mV P max ). For example, evaluated on the Market-1501 dataset, DenseNet-121 has a better mAP score than MLFN does, but it does not have a better mV P max score. It shows that our metric introduce more focus on the different aspects of evaluated methods.
3) Figure 3 shows that different methods touch the top mReP max scores at different thresholds. It means that when we want to adopt a method to some applications, we should set a proper threshold to obtain its best performances. Here, we list the threshold τ max along with the mReP max in Table 3. Table 3 shows that the supervised method AGW and unsupervised method SpCL obtain the best mReP max scores on both person re-ID datasets, respectively. A high mReP max score indicates that within the returned samples, high ranked samples are very likely to be the targets. mReP max demonstrates that it has powerful competitiveness in the verification task and ensures the reliability of the retrieval. 5) Both Table 3 and Table 4 show that the pros and cons of results among different methods are similar by existing metrics and our metrics. Since the retrieval has a very strong relationship with verification, and also contributes a lot to the GOM metric. Thus, it is reasonable that existing methods and proposed GOM have high global similarity, and some local variations.

4)
6) The BagTricks method gets the highest M REP score. As discussed above, to get a high mReP max score, we need to know the best threshold of the selected method. If we do not know this threshold, BagTricks is the best choice.
7) The BagTricks method also gets the highest M F R score. Thus, although it performs very well in the closedworld re-ID, it does not fit the open-set re-ID well. Note that existing metrics cannot provide this suggestion. Summary. 1) Most of the current methods have little overall difference. If an analysis is made from the perspectives of re-ID, the performance evaluation cannot be made from the value of the mAP alone. Our GOM can provide comprehensive performance evaluation from an intuitive perspective (mReP and mF R curves) and objective perspective (mV P max , mReP max , M REP , and M F R). 2) Different sub-metrics show the importance of different aspects, for instance, if the verification is the most important for an application, a method with the best mV P max should be selected; if verification and retrieval are both considered and balanced, a method with the best mReP max should be selected (its threshold should be recorded); if the threshold cannot be set fixed, a method with the best M REP should be selected; if we need to consider the open-set problem, and do not like so many noise influence the application, a method with the best M F R should be selected.

Human visual assessment study
In this subsection, we perform a user study in order to determine how designed metrics align with the human judgment of re-ID quality. We invite 13 participants to conduct a user study. Each participant is asked to review the results of 21 hard queries generated by five typical re-ID methods on the Market-1501 dataset. The results have two forms, one is a ranking list for each query, the other is a batch of returned samples by a threshold of mReP max , which are all considered as the targets. During browsing the ranking list, the participant search out the targets in each ranking list, and the cost time is recorded. After reviewing each batch of returned samples, the participant gives a rating to five evaluated methods (1-5, 5 is the highest score to indicate the best method). Finally, we have the average ratings for different methods and the average time cost of searching out targets in each ranking list. The results are recorded in Table 5. Since we consider that re-ID quality consists of quality of verification and quality of retrieval. Our mReP max and M REP sub-metrics both consider quality verification and retrieval. For the user part, our average ratings and average search time reflect the quality verification and retrieval, respectively. Thus, Table 5 shows that the designed metrics well align with the human judgment of re-ID quality.

Conclusion
In this paper, we analyzed the current re-ID metrics based on three types of errors and showed their limitations. Then a new metric, Genuine Open-set re-ID Metric (GOM), has been proposed for the re-ID task. GOM shows the comprehensive performance of re-ID methods from the respective verification task and retrieval task. We consider that existing metrics are designed for retrieval task, essentially, not suitable for re-ID. With GOM, researchers can evaluate re-ID methods in a thorough way, and developers are able to select suitable algorithms to build their application. We encourage the re-ID community to consider this metric in future algorithms evaluations and comparisons.
Prospects. Note that even our proposed GOM metric is comprehensive, there are still some directions worth noting. 1) All existing methods cannot evaluate a dynamic gallery system or an online application. It is because the gallery size and the outputs always change. Thus, designing a metric for the dynamic gallery system is a prospective direction. 2) We still didn't consider the situations that the results are separated, fragmented, or even protected due to the privacy concern. How to design a metric to address these issues is also a direction that should be discussed. In the future, we will continuously investigate how to make a more comprehensive evaluation for re-ID.