Question: represent the scientific article to find a short tldr:

Question: Nowadays, deep neural networks (DNNs) have become the main instrument for machine learning tasks within a wide range of domains, including vision, NLP, and speech. Meanwhile, in an important case of heterogenous tabular data, the advantage of DNNs over shallow counterparts remains questionable. In particular, there is no sufficient evidence that deep learning machinery allows constructing methods that outperform gradient boosting decision trees (GBDT), which are often the top choice for tabular problems. In this paper, we introduce Neural Oblivious Decision Ensembles (NODE), a new deep learning architecture, designed to work with any tabular data. In a nutshell, the proposed NODE architecture generalizes ensembles of oblivious decision trees, but benefits from both end-to-end gradient-based optimization and the power of multi-layer hierarchical representation learning. With an extensive experimental comparison to the leading GBDT packages on a large number of tabular datasets, we demonstrate the advantage of the proposed NODE architecture, which outperforms the competitors on most of the tasks. We open-source the PyTorch implementation of NODE and believe that it will become a universal framework for machine learning on tabular data. The recent rise of deep neural networks (DNN) ed in a substantial breakthrough for a large number of machine learning tasks in computer vision, natural language processing, speech recognition, reinforcement learning . Both gradient-based optimization via backpropagation and hierarchical representation learning appear to be crucial in increasing the performance of machine learning for these problems by a large margin. While the superiority of deep architectures in these domains is undoubtful, machine learning for tabular data still did not fully benefit from the DNN power. Namely, the state-of-the-art performance in problems with tabular heterogeneous data is often achieved by "shallow" models, such as gradient boosted decision trees (GBDT) (; ; ;). While the importance of deep learning on tabular data is recognized by the ML community, and many works address this problem (; ; ; ;), the proposed DNN approaches do not consistently outperform the state-of-the-art shallow models by a notable margin. In particular, to the best of our knowledge, there is still no universal DNN approach that was shown to systematically outperform the leading GBDT packages (e.g., XGBoost ). As additional evidence, a large number of Kaggle ML competitions with tabular data are still won by the shallow GBDT methods . Overall, at the moment, there is no dominant deep learning solution for tabular data problems, and we aim to reduce this gap by our paper. We introduce Neural Oblivious Decision Ensembles (NODE), a new DNN architecture, designed to work with tabular problems. The NODE architecture is partially inspired by the recent CatBoost package , which was shown to provide state-of-the-art performance on a large number of tabular datasets. In a nutshell, CatBoost performs gradient boosting on oblivious decision trees (decision tables) , which makes inference very efficient, and the method is quite resistant to overfitting. In its essence, the proposed NODE architecture generalizes CatBoost, making the splitting feature choice and decision tree routing differentiable. As a , the NODE architecture is fully differentiable and could be incorporated in any computational graph of existing DL packages, such as TensorFlow or PyTorch. Furthermore, NODE allows constructing multi-layer architectures, which resembles "deep" GBDT that is trained end-to-end, which was never proposed before. Besides the usage of oblivious decision tables, another important design choice is the recent entmax transformation , which effectively performs a "soft" splitting feature choice in decision trees inside the NODE architecture. As discussed in the following sections, these design choices are critical to obtain state-of-the-art performance. In a large number of experiments, we compare the proposed approach with the leading GBDT implementations with tuned hyperparameters and demonstrate that NODE outperforms competitors consistently on most of the datasets. Overall, the main contributions of our paper can be summarized as follows: 1. We introduce a new DNN architecture for machine learning on tabular data. To the best of our knowledge, our method is the first successful example of deep architectures that substantially outperforms leading GBDT packages on tabular data. 2. Via an extensive experimental evaluation on a large number of datasets, we show that the proposed NODE architecture outperforms existing GBDT implementations. 3. The PyTorch implementation of NODE is available online 1. The rest of the paper is organized as follows. In Section 2 we review prior work relevant to our method. The proposed Neural Oblivious Decision Ensembles architecture is described in Section 3 and experimentally evaluated in Section 4. Section 5 concludes the paper. In this section, we briefly review the main ideas from prior work that are relevant to our method. The state-of-the-art for tabular data. Ensembles of decision trees, such as GBDT or random forests , are currently the top choice for tabular data problems. Currently, there are several leading GBDT packages, such as XGBoost , LightGBM , CatBoost , which are widely used by both academicians and ML practitioners. While these implementations vary in details, on most of the tasks their performances do not differ much (; Anghel et al.). The most important distinction of CatBoost is that it uses oblivious decision trees (ODTs) as weak learners. As ODTs are also an important ingredient of our NODE architecture, we discuss them below. Oblivious Decision Trees. An oblivious decision tree is a regular tree of depth d that is constrained to use the same splitting feature and splitting threshold in all internal nodes of the same depth. This constraint essentially allows representing an ODT as a table with 2 d entries, corresponding to all possible combinations of d splits . Of course, due to the constraints above, ODTs are significantly weaker learners compared to unconstrained decision trees. However, when used in an ensemble, such trees are less prone to overfitting, which was shown to synergize well with gradient boosting . Furthermore, the inference in ODTs is very efficient: one can compute d independent binary splits in parallel and return the appropriate table entry. In contrast, non-oblivious decision trees require evaluating d splits sequentially. Differentiable trees. The significant drawback of tree-based approaches is that they usually do not allow end-to-end optimization and employ greedy, local optimization procedures for tree construction. Thus, they cannot be used as a component for pipelines, trained in an end-to-end fashion. To address this issue, several works propose to "soften" decision functions in the internal tree nodes to make the overall tree function and tree routing differentiable. In our work, we advocate the usage of the recent entmax transformation to "soften" decision trees. We confirm its advantages over the previously proposed approaches in the experimental section. Entmax. The key building block of our model is the entmax transformation , which maps a vector of real-valued scores to a discrete probability distribution. This transformation generalizes the traditional softmax and its sparsity-enforcing alternative sparsemax , which has already received significant attention in a wide range of applications: probabilistic inference, topic modeling, neural attention (; ;). The entmax is capable to produce sparse probability distributions, where the majority of probabilities are exactly equal to 0. In this work, we argue that entmax is also an appropriate inductive bias in our model, which allows differentiable split decision construction in the internal tree nodes. Intuitively, entmax can learn splitting decisions based on a small subset of data features (up to one, as in classical decision trees), avoiding undesired influence from others. As an additional advantage, using entmax for feature selection allows for computationally efficient inference using the sparse pre-computed choice vectors as described below in Section 3. Multi-layer non-differentiable architectures. Another line of work (; ;) promotes the construction of multi-layer architectures from nondifferentiable blocks, such as random forests or GBDT ensembles. For instance, propose to use stacking of several random forests, which are trained separately. In recent work, introduces the multi-layer GBDTs and proposes a training procedure that does not require each layer component to be differentiable. While these works report marginal improvements over shallow counterparts, they lack the capability for end-to-end training, which could in inferior performance. In contrast, we argue that end-to-end training is crucial and confirm this claim in the experimental section. Specific DNN for tabular data. While a number of prior works propose architectures designed for tabular data , they mostly do not compare with the properly tuned GBDT implementations, which are the most appropriate baselines. The recent preprint reports the marginal improvement over GBDT with default parameters, but in our experiments, the baseline performance is much higher. To the best of our knowledge, our approach is the first to consistently outperform the tuned GBDTs over a large number of datasets. We introduce the Neural Oblivious Decision Ensemble (NODE) architecture with a layer-wise structure similar to existing deep learning models. In a nutshell, our architecture consists of differentiable oblivious decision trees (ODT) that are trained end-to-end by backpropagation. We describe our implementation of the differentiable NODE layer in Section 3.1, the full model architecture in Section 3.2, and the training and inference procedures in section 3.3. The core building block of our model is a Neural Oblivious Decision Ensemble (NODE) layer. The layer is composed of m differentiable oblivious decision trees (ODTs) of equal depth d. As an input, all m trees get a common vector x ∈ R n, containing n numeric features. Below we describe a design of a single differentiable ODT.... Figure 1: The single ODT inside the NODE layer. The splitting features and the splitting thresholds are shared across all the internal nodes of the same depth. The output is a sum of leaf responses scaled by the choice weights. In its essence, an ODT is a decision table that splits the data along d splitting features and compares each feature to a learned threshold. Then, the tree returns one of the 2 d possible responses, corresponding to the comparisons . Therefore, each ODT is completely determined by its splitting. In this notation, the tree output is defined as: where 1(·) denotes the Heaviside function. To make the tree output differentiable, we replace the splitting feature choice f i and the comparison operator 1(f i (x) − b i ) by their continuous counterparts. There are several existing approaches that can be used for modelling differentiable choice functions in decision trees, for instance, REINFORCE or Gumbel-softmax . However, these approaches typically require long training time, which can be crucial in practice. Instead, we propose to use the α-entmax function as it is able to learn sparse choices, depending only on a few features, via standard gradient descent. This function is a generalization of softmax in its variational form: is Shannon entropy. We can define α-entmax by replacing H(p) with Tsallis α-entropy 2. The choice function is hence replaced by a weighted sum of features, with weights computed as α-entmax (α=1.5) over the learnable feature selection matrix F ∈ R d×n: Similarly, we relax the Heaviside function 1(f i (x) − b i ) as a two-class entmax, which we denote as, where b i and τ i are learnable parameters for thresholds and scales respectively. Based on the c i (x) values, we define a "choice" tensor C ∈ R 2 × 2 × 2 d of the same size as the response tensor R by computing the outer product of all c i: The final prediction is then computed as a weighted linear combination of response tensor entries R with weights from the entries of choice tensor C: Note, that this relaxation equals to the classic non-differentiable ODT h(x) iff both feature selection and threshold functions reach one-hot state, i.e. entmax always returns non-zero weights for a single feature and c i always return exactly zeros or ones. Finally, the output of the NODE layer is composed as a concatenation of the outputs of m individual trees ĥ 1 (x),...,ĥ m (x). Multidimensional tree outputs. In the description above, we assumed that tree outputs are onedimensionalĥ(x) ∈ R. For classification problems, where NODE predicts probabilities of each class, we use multidimensional tree outputsĥ(x) ∈ R |C|, where |C| is a number of classes. The NODE layer, described above, can be trained alone or within a complex structure, like fullyconnected layers that can be organized into the multi-layer architectures. In this work, we introduce a new architecture, following the popular DenseNet Figure 2: The NODE architecture, consisting of densely connected NODE layers. Each layer contains several trees whose outputs are concatenated and serve as input for the subsequent layer. The final prediction is obtained by averaging the outputs of all trees from all the layers. Similar to DenseNet, our architecture is a sequence of k NODE layers (see Section 3.1), where each layer uses a concatenation of all previous layers as its input. The input layer 0 of this architecture corresponds to the input features x, accessible by all successor layers. Due to such a design, our architecture is capable to learn both shallow and deep decision rules. A single tree on i-th layer can rely on chains of up to i − 1 layer outputs as features, allowing it to capture complex dependencies. The ing prediction is a simple average of all decision trees from all layers. Note, in the multi-layer architecture described above, tree outputsĥ(x) from early layers are used as inputs for subsequent layers. Therefore, we do not restrict the dimensionality ofĥ(x) to be equal to the number of classes, and allow it to have an arbitrary dimensionality l, which correspond to When averaging the predictions from all layers, only first |C| coordinates ofĥ(x) are used for classification problems and the first one for regression problems. Overall, l is an additional hyperparameter with typical values in. Here we summarize the details of our training protocol. Data preprocessing. First, we transform each data feature to follow a normal distribution via quantile transform 3. In experiments, we observed that this step was important for stable training and faster convergence. Initialization. Before training, we perform the data-aware initialization to obtain a good initial parameter values. In particular, we initialize the feature selection matrix uniformly F ij ∼ U, while the thresholds b are initialized with random feature values f i (x) observed in the first data batch. The scales τ i are initialized in such a way that all the samples in the first batch belong to the linear region of σ α, and hence receive nonzero gradients. Finally, the response tensor entries are initialized with the standard normal distribution R[i 1, . . ., i d] ∼ N. We jointly optimize all model parameters: F, b, R. In this work, we experimented with traditional objective functions (cross-entropy for classification and mean squared error for regression), but any differentiable objective can be used as well. As an optimization method, we use the recent QuasiHyperbolic Adam with parameters recommended in the original paper . We also average the model parameters over c = 5 consecutive checkpoints and pick the optimal stopping point on the hold-out validation dataset. Inference. During training, a significant fraction of time is spent computing the entmax function and multiplying the choice tensor. Once the model is trained, one can pre-compute entmax feature selectors and store them as a sparse vector (e.g., in coordinate (coo) format), making inference more efficient. In this section, we report the of a comparison between our approach and the leading GBDT packages. We also provide several ablation studies that demonstrate the influence of each design choice in the proposed NODE architecture. As our main experiments, we compare the proposed NODE architecture with two state-of-the-art GBDT implementations on a large number of datasets. In all the experiments we set α parameter in the entmax transformation to 1.5. All other details of the comparison protocol are described below. Datasets. We perform most of the experiments on six open-source tabular datasets from different domains: Epsilon, YearPrediction, Higgs, Microsoft, Yahoo, Click. The detailed description of the datasets is available in appendix. All the datasets provide train/test splits, and we used 20% samples from the train set as a validation set to tune the hyperparameters. For each dataset, we fix the train/val/test splits for a fair comparison. For the classification datasets (Epsilon, Higgs, Click), we minimize cross-entropy loss and report the classification error. For the regression and ranking datasets (YearPrediction, Microsoft, Yahoo), we minimize and report mean squared error (which corresponds to the pointwise approach to learning-to-rank). Methods. We compare the proposed NODE architecture to the following baselines: • Catboost. The recent GBDT implementation that uses oblivious decision trees as weak learners. We use the open-source implementation, provided by the authors. • XGBoost. The most popular GBDT implementation widely used in machine learning competitions . We use the open-source implementation, provided by the authors. • FCNN. Deep neural network, consisting of several fully-connected layers with ReLU nonlinearity layers . Regimes. We perform comparison in two following regimes that are the most important in practice: • Default hyperparameters. In this regime, we compare the methods as easy-to-tune toolkits that could be used by a non-professional audience. Namely, here we do not tune hyperparameters and use the default ones provided by the GBDT packages. The only tunable parameter here is a number of trees (up to 2048) in CatBoost/XGBoost, which is set based on the validation set. We do not compare with FCNN in this regime, as it typically requires much tuning, and we did not find the set of parameters, appropriate for all datasets. The default architecture in our model contains only a single layer with 2048 decision trees of depth six. Both of these hyperparameters were inherited from the CatBoost package settings for oblivious decision trees. With these parameters, the NODE architecture is shallow, but it still benefits from end-to-end training via back-propagation. • Tuned hyperparameters. In this regime, we tune the hyperparameters for both NODE and the competitors on the validation subsets. The optimal configuration for NODE contains between two and eight NODE layers, while the total number of trees across all the layers does not exceed 2048. The details of hyperparameter optimization are provided in appendix. The of the comparison are summarized in Table 1 and Table 2. For all methods, we report mean performance and standard deviations computed over ten runs with different random seeds. Several key observations are highlighted below: 1. With default hyperparameters, the proposed NODE architecture consistently outperforms both CatBoost and XGBoost on all datasets. The advocate the usage of NODE as a handy tool for machine learning on tabular problems. 2. With tuned hyperparameters, NODE also outperforms the competitors on most of the tasks. Two exceptions are the Yahoo and Microsoft datasets, where tuned XGBoost provides the highest performance. Given the large advantage of XGBoost over CatBoost on Yahoo, we speculate that the usage of oblivious decision trees is an inappropriate inductive bias for this dataset. This implies that NODE should be extended to non-oblivious trees, which we leave for future work. Table 2: The comparison of NODE with both shallow and deep counterparts with hyperparameters tuned for optimal performance. The are computed over ten runs with different random seeds. 3. In the regime with tuned hyperparameters on some datasets FCNN outperforms GBDT, while on others GBDT is superior. Meanwhile, the proposed NODE architecture appears to be a universal instrument, providing the highest performance on most of the tasks. For completeness we also aimed to compare to previously proposed architectures for deep learning on tabular data. Unfortunately, many works did not publish the source code. We were only able to perform a partial comparison with mGBDT and DeepForest , which source code is available. For both baselines, we use the implementations, provided by the authors, and tune the parameters on the validation set. Note, that the DeepForest implementation is available only for classification problems. Moreover, both implementations do not scale well, and for many datasets, we obtained Out-Of-Memory error (OOM). On datasets in our experiments it turns out that properly tuned GBDTs outperform both and . In this section, we analyze the key architecture components that define our model. Choice functions. Constructing differentiable decision trees requires a function that selects items from a set. Such function is required for both splitting feature selection and decision tree routing. We experimented with four possible options, each having different implications: • Softmax learns dense decision rules where all items have nonzero weights; • Gumbel-Softmax learns to stochastically sample a single element from a set; • Sparsemax learns sparse decision rules, where only a few items have nonzero weights; • Entmax generalizes both sparsemax and softmax; it is able to learn sparse decision rules, but is smoother than sparsemax, being more appropriate for gradient-based optimization. In comparison α parameter was set to 1.5. We experimentally compare the four options above with both shallow and deep architectures in Table 3. We use the same choice function for both feature selection and tree routing across all experiments. In Gumbel-Softmax, we replaced it with hard argmax one-hot during inference. The clearly show that Entmax with α=1.5 outperforms the competitors across all experiments. First, Table 3 demonstrates that sparsemax and softmax are not universal choice functions. For instance, on the YearPrediction dataset, sparsemax outperforms softmax, while on the Epsilon dataset softmax is superior. In turn, entmax provides great empirical performance across all datasets. Another observation is that Gumbel-Softmax is unable to learn deep architectures with both constant and annealed temperature schedules. This behavior is probably caused by the stochasticity of Gumbel-Softmax and the responses on the former layers are too noisy to produce useful features for the latter layers. . Namely, for 10.000 objects from the Higgs dataset we randomly shuffle the values of each feature (original or learnt on some NODE layer) and compute the increase in the classification error. Then for each layer, we split feature importance values into seven equal bins and calculate the total feature importance of each bin, shown on Figure 3 (left-top). We discovered that the features from the first layer are used the most, with feature importances decreasing with depth. This figure shows that deep layers are able to produce important features, even though earlier layers have an advantage because of the DenseNet architecture. Next, we estimated the mean absolute contribution of individual trees to the final response, reported on Figure 3 (left-bottom). One can see the reverse trend, deep trees tend to contribute more to the final response. Figure 3 (right) clearly shows that there is anticorrelation of feature importances and contributions in the final response, which implies that the main role of ealier layers is to produce informative features, while the latter layers mostly use them for accurate prediction. Training/Inference runtime. Finally, we compare the NODE runtime to the timings of the stateof-the-art GBDT implementations. In Table 4 we report the training and inference time for million of objects from the YearPrediction dataset. In this experiment, we evaluate ensembles of 1024 trees of depth six with all other parameters set to their default values. Our GPU setup has a single 1080Ti GPU and 2 CPU cores. In turn, our CPU setup has a 28-core Xeon E5-2660 v4 processor (which costs almost twice as much as the GPU Table 4 : Training and inference runtime for models with 1024 trees of depth six on the YearPrediction dataset, averaged over five runs. Both training and inference of eight-layer NODE architecture on GPU are on par with shallow counterparts of the same total number of trees in an ensemble. In this paper, we introduce a new DNN architecture for deep learning on heterogeneous tabular data. The architecture is differentiable deep GBDTs, trained end-to-end via backpropagation. In extensive experiments, we demonstrate the advantages of our architecture over existing competitors with the default and tuned hyperparameters. A promising research direction is incorporating the NODE layer into complex pipelines trained via back-propagation. For instance, in multi-modal problems, the NODE layer could be employed as a way to incorporate the tabular data, as CNNs are currently used for images, or RNNs are used for sequences. library to optimize Catboost, XGBoost, and FCNN hyperparameters. For each method, we perform 50 steps of Tree-structured Parzen Estimator (TPE) optimization algorithm. As a final configuration, we choose the set of hyperparameters, corresponding to the smallest loss on the validation set. On each iteration of Hyperopt, the number of trees was set based on the validation set, with maximal trees count set to 2048. Below is the list of hyperparameters and their search spaces for Catboost.

We propose a new DNN architecture for deep learning on tabular data

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the text to find a very short summary (1-2 sentences max):

Question: Person re-identification (re-ID) aims at identifying the same persons' images across different cameras. However, domain diversities between different datasets pose an evident challenge for adapting the re-ID model trained on one dataset to another one. State-of-the-art unsupervised domain adaptation methods for person re-ID transferred the learned knowledge from the source domain by optimizing with pseudo labels created by clustering algorithms on the target domain. Although they achieved state-of-the-art performances, the inevitable label noise caused by the clustering procedure was ignored. Such noisy pseudo labels substantially hinders the model's capability on further improving feature representations on the target domain. In order to mitigate the effects of noisy pseudo labels, we propose to softly refine the pseudo labels in the target domain by proposing an unsupervised framework, Mutual Mean-Teaching (MMT), to learn better features from the target domain via off-line refined hard pseudo labels and on-line refined soft pseudo labels in an alternative training manner. In addition, the common practice is to adopt both the classification loss and the triplet loss jointly for achieving optimal performances in person re-ID models. However, conventional triplet loss cannot work with softly refined labels. To solve this problem, a novel soft softmax-triplet loss is proposed to support learning with soft pseudo triplet labels for achieving the optimal domain adaptation performance. The proposed MMT framework achieves considerable improvements of 14.4%, 18.2%, 13.1% and 16.4% mAP on Market-to-Duke, Duke-to-Market, Market-to-MSMT and Duke-to-MSMT unsupervised domain adaptation tasks.: Person image A 1 and A 2 belong to the same identity while B with similar appearance is from another person. However, clustering-generated pseudo labels in state-of-the-art Unsupervised Domain Adaptation (UDA) methods contain much noise that hinders feature learning. We propose pseudo label refinery with on-line refined soft pseudo labels to effectively mitigate the influence of noisy pseudo labels and improve UDA performance on person re-ID. To effectively address the problem of noisy pseudo labels in clustering-based UDA methods (; b; (Figure 1), we propose an unsupervised Mutual Mean-Teaching (MMT) framework to effectively perform pseudo label refinery by optimizing the neural networks under the joint supervisions of off-line refined hard pseudo labels and on-line refined soft pseudo labels. Specifically, our proposed MMT framework provides robust soft pseudo labels in an on-line peer-teaching manner, which is inspired by the teacher-student approaches (; b) to simultaneously train two same networks. The networks gradually capture target-domain data distributions and thus refine pseudo labels for better feature learning. To avoid training error amplification, the temporally average model of each network is proposed to produce reliable soft labels for supervising the other network in a collaborative training strategy. By training peer-networks with such on-line soft pseudo labels on the target domain, the learned feature representations can be iteratively improved to provide more accurate soft pseudo labels, which, in turn, further improves the discriminativeness of learned feature representations. The classification and triplet losses are commonly adopted together to achieve state-of-the-art performances in both fully-supervised and unsupervised (b; person re-ID models. However, the conventional triplet loss cannot work with such refined soft labels. To enable using the triplet loss with soft pseudo labels in our MMT framework, we propose a novel soft softmax-triplet loss so that the network can benefit from softly refined triplet labels. The introduction of such soft softmax-triplet loss is also the key to the superior performance of our proposed framework. Note that the collaborative training strategy on the two networks is only adopted in the training process. Only one network is kept in the inference stage without requiring any additional computational or memory cost. The contributions of this paper could be summarized as three-fold. We propose to tackle the label noise problem in state-of-the-art clustering-based UDA methods for person re-ID, which is mostly ignored by existing methods but is shown to be crucial for achieving superior final performance. The proposed Mutual Mean-Teaching (MMT) framework is designed to provide more reliable soft labels. Conventional triplet loss can only work with hard labels. To enable training with soft triplet labels for mitigating the pseudo label noise, we propose the soft softmax-triplet loss to learn more discriminative person features. The MMT framework shows exceptionally strong performances on all UDA tasks of person re-ID. Compared with state-of-the-art methods, it leads to significant improvements of 14.4%, 18.2%, 13.4%, 16.4% mAP on Market-to-Duke, Duke-to-Market, Market-to-MSMT, Duke-to-MSMT re-ID tasks. Unsupervised domain adaptation (UDA) for person re-ID. UDA methods have attracted much attention because their capability of saving the cost of manual annotations. There are three main categories of methods. The first category of clustering-based methods maintains state-of-the-art performance to date. proposed to alternatively assign labels for unlabeled training samples and optimize the network with the generated targets. proposed a bottom-up clustering framework with a repelled loss. introduced to assign hard pseudo labels for both global and local features. However, the training of the neural network was substantially hindered by the noise of the hard pseudo labels generated by clustering algorithms, which was mostly ignored by existing methods. The second category of methods learns domain-invariant features from style-transferred source-domain images. SPGAN and PTGAN transformed source-domain images to match the image styles of the target domain while maintaining the original person identities. The style-transferred images and their identity labels were then used to fine-tune the model. HHL learned camera-invariant features with camera style transferred images. However, the retrieval performances of these methods deeply relied on the image generation quality, and they did not explore the complex relations between different samples in the target domain. The third category of methods attempts on optimizing the neural networks with soft labels for target-domain samples by computing the similarities with reference images or features. ENC assigned soft labels by saving averaged features with an exemplar memory module. MAR conducted multiple soft-label learning by comparing with a set of reference persons. However, the reference images and features might not be representative enough to generate accurate labels for achieving advanced performances. Generic domain adaptation methods for close-set recognition. Generic domain adaptation methods learn features that can minimize the differences between data distributions of source and target domains. Adversarial learning based methods (a; ; ; ;) adopted a domain classifier to dispel the discriminative domain information from the learned features in order to reduce the domain gap. There also exist methods (; ; ; ;) that minimize the Maximum Mean Discrepancy (MMD) loss between source-and target-domain distributions. However, these methods assume that the classes on different domains are shared, which is not suitable for unsupervised domain adaptation on person re-ID. Teacher-student models have been widely studied in semi-supervised learning methods and knowledge/model distillation methods. The key idea of teacher-student models is to create consistent training supervisions for labeled/unlabeled data via different models' predictions. Temporal ensembling maintained an exponential moving average prediction for each sample as the supervisions of the unlabeled samples, while the mean-teacher model averaged model weights at different training iterations to create the supervisions for unlabeled samples. Deep mutual learning (b) adopted a pool of student models instead of the teacher models by training them with supervisions from each other. However, existing methods with teacher-student mechanisms are mostly designed for close-set recognition problems, where both labeled and unlabeled data share the same set of class labels and could not be directly utilized on unsupervised domain adaptation tasks of person re-ID. Generic methods for handling noisy labels can be classified into four categories. Loss correction methods (; ;) tried to model the noise transition matrix, however, such matrix is hard to estimate in real-world tasks, e.g. unsupervised person re-ID with noisy pseudo labels obtained via clustering algorithm. (; ;) attempted to correct the noisy labels directly, while the clean set required by such methods limits their generalization on real-world applications. Noise-robust methods designed robust loss functions against label noises, for instance, Mean Absolute Error (MAE) loss , Generalized Cross Entropy (GCE) loss and Label Smoothing Regularization (LSR) . However, these methods did not study how to handle the triplet loss with noisy labels, which is crucial for learning discriminative feature representations on person re-ID. The last kind of methods which focused on refining the training strategies is mostly related to our method. Co-teaching trained two collaborative networks and conducted noisy label detection by selecting on-line clean data for each other, Co-mining further extended this method on the face recognition task with a re-weighting function for Arc-Softmax loss . However, the above methods are not designed for the open-set person re-ID task and could not achieve state-of-the-art performances under the more challenge unsupervised settings. We propose a novel Mutual Mean-Teaching (MMT) framework for tackling the problem of noisy pseudo labels in clustering-based Unsupervised Domain Adaptation (UDA) methods. The label noise has important impacts to the domain adaptation performance but was mostly ignored by those methods. Our key idea is to conduct pseudo label refinery in the target domain by optimizing the neural networks with off-line refined hard pseudo labels and on-line refined soft pseudo labels in a collaborative training manner. In addition, the conventional triplet loss cannot properly work with soft labels. A novel soft softmax-triplet loss is therefore introduced to better utilize the softly refined pseudo labels. Both the soft classification loss and the soft softmax-triplet loss work jointly to achieve optimal domain adaptation performances. 3.1 CLUSTERING-BASED UDA METHODS REVISIT State-of-the-art UDA methods (; b; follow a similar general pipeline. They generally pre-train a deep neural network F (·|θ) on the source domain, where θ denotes current network parameters, and the network is then transferred to learn from the images in the target domain. The source-domain images' and target-domain images' features encoded by the network are denoted as {F (x The network parameters θ and a learnable target-domain classifier C t: f t → {1, · · ·, M t} are then optimized with respect to an identity classification (crossentropy) loss L t id (θ) and a triplet loss (where || · || denotes the L 2 -norm distance, subscripts i,p and i,n indicate the hardest positive and hardest negative feature index in each mini-batch for the sample x t i, and m = 0.5 denotes the triplet distance margin. Such two operations, pseudo label generation by clustering and feature learning with pseudo labels, are alternated until the training converges. However, the pseudo labels generated in step inevitably contain errors due to the imperfection of features as well as the errors of the clustering algorithms, which hinder the feature learning in step. To mitigate the pseudo label noise, we propose the Mutual Mean-Teaching (MMT) framework together with a novel soft softmax-triplet loss to conduct the pseudo label refinery. 3.2.1 SUPERVISED PRE-TRAINING FOR SOURCE DOMAIN UDA task on person re-ID aims at transferring the knowledge from a pre-trained model on the source domain to the target domain. A deep neural network is first pre-trained on the source domain. Given the training data D s, the network is trained to model a feature transformation function F (·|θ) that transforms each input sample x s i into a feature representation F (x s i |θ). Given the encoded features, the identification classifier C s outputs an M s -dimensional probability vector to predict the identities in the source-domain training set. The neural network is trained with a classification loss L s id (θ) and a triplet loss L s tri (θ) to separate features belonging to different identities. The overall loss is therefore calculated as where L Overall framework of the proposed Mutual Mean-Teaching (MMT) with two collaborative networks jointly optimized under the supervisions of off-line refined hard pseudo labels and on-line refined soft pseudo labels. A soft identity classification loss and a novel soft softmax-triplet loss are adopted. (c) One of the average models with better validated performance is adopted for inference as average models perform better than models with current parameters. Our proposed MMT framework is based on the clustering-based UDA methods with off-line refined hard pseudo labels as introduced in Section 3.1, where the pseudo label generation and refinement are conducted alternatively. However, the pseudo labels generated in this way are hard (i.e., they are always of 100% confidences) but noisy. In order to mitigate the pseudo label noise, apart from the off-line refined hard pseudo labels, our framework further incorporates on-line refined soft pseudo labels (i.e., pseudo labels with < 100% confidences) into the training process. Our MMT framework generates soft pseudo labels by collaboratively training two same networks with different initializations. The overall framework is illustrated in Figure 2 (b). The pseudo classes are still generated the same as those by existing clustering-based UDA methods, where each cluster represents one class. In addition to the hard and noisy pseudo labels, our two collaborative networks also generate on-line soft pseudo labels by network predictions for training each other. The intuition is that, after the networks are trained even with hard pseudo labels, they can roughly capture the training data distribution and their class predictions can therefore serve as soft class labels for training. However, such soft labels are generally not perfect because of the training errors and noisy hard pseudo labels in the first place. To avoid two networks collaboratively bias each other, the past temporally average model of each network instead of the current model is used to generate the soft pseudo labels for the other network. Both off-line hard pseudo labels and on-line soft pseudo labels are utilized jointly to train the two collaborative networks. After training, only one of the past average models with better validated performance is adopted for inference (see Figure 2 (c)). We denote the two collaborative networks as feature transformation functions F (·|θ 1) and F (·|θ 2), and denote their corresponding pseudo label classifiers as C t 1 and C t 2, respectively. To simultaneously train the coupled networks, we feed the same image batch to the two networks but with separately random erasing, cropping and flipping. Each target-domain image can be denoted by x t i and x t i for the two networks, and their pseudo label confidences can be predicted as C t 1 (F (x t i |θ 1)) and C t 2 (F (x t i |θ 2)). One naïve way to train the collaborative networks is to directly utilize the above pseudo label confidence vectors as the soft pseudo labels for training the other network. However, in such a way, the two networks' predictions might converge to equal each other and the two networks lose their output independences. The classification errors as well as pseudo label errors might be amplified during training. In order to avoid error amplification, we propose to use the temporally average model of each network to generate reliable soft pseudo labels for supervising the other network. Specifically, the parameters of the temporally average models of the two networks at current iteration T are denoted as respectively, which can be calculated as where indicate the temporal average parameters of the two networks in the previous iteration (T −1), the initial temporal average parameters are and α is the ensembling momentum to be within the range. The robust soft pseudo label supervisions are then generated by the two temporal average models as respectively. The soft classification loss for optimizing θ 1 and θ 2 with the soft pseudo labels generated from the other network can therefore be formulated as The two networks' pseudo-label predictions are better dis-related by using other network's past average model to generate supervisions and can therefore better avoid error amplification. Generalizing classification cross-entropy loss to work with soft pseudo labels has been well studied , (Müller et al., 2019). However, optimizing triplet loss with soft pseudo labels poses a great challenge as no previous method has investigated soft labels for triplet loss. For tackling the difficulty, we propose to use softmax-triplet loss, whose hard version is formulated as where Here L bce (·, ·) denotes the binary cross-entropy loss, F (x and its positive sample x t i,p to measure their similarity, and "1" denotes the ground-truth that the positive sample x t i,p should be closer to the sample x t i than its negative sample x t i,n . Given the two collaborative networks, we can utilize the one network's past temporal average model to generate soft triplet labels for the other network with the proposed soft softmax-triplet loss, where are the soft triplet labels generated by the two networks' past temporally average models. Such soft triplet labels are fixed as training supervisions. By adopting the soft softmax-triplet loss, our MMT framework overcomes the limitation of hard supervisions by the conventional triple loss (equation 2). It can be successfully trained with soft triplet labels, which are shown to be important for improving the domain adaptation performance in our experiments. Note that such a softmax-triplet loss was also studied in (a). However, it has never been used to generate soft labels and was not designed to work with soft pseudo labels before. Our proposed MMT framework is trained with both off-line refined hard pseudo labels and on-line refined soft pseudo labels. The overall loss function L(θ 1, θ 2) simultaneously optimizes the coupled networks, which combines equation 1 We evaluate our proposed MMT on three widely-used person re-ID datasets, i.e., Market-1501 , DukeMTMC-reID , and MSMT17. The Market-1501 dataset consists of 32,668 annotated images of 1,501 identities shot from 6 cameras in total, for which 12,936 images of 751 identities are used for training and 19,732 images of 750 identities are in the test set. DukeMTMC-reID contains 16,522 person images of 702 identities for training, and the remaining images out of another 702 identities for testing, where all images are collected from 8 cameras. MSMT17 is the most challenging and large-scale dataset consisting of 126,441 bounding boxes of 4,101 identities taken by 15 cameras, for which 32,621 images of 1,041 identities are spitted for training. For evaluating the domain adaptation performance of different methods, four domain adaptation tasks are set up, i.e., Duke-to-Market, Market-to-Duke, Duke-to-MSMT and Market-to-MSMT, where only identity labels on the source domain are provided. Mean average precision (mAP) and CMC top-1, top-5, top-10 accuracies are adopted to evaluate the methods' performances. 4.2.1 TRAINING DATA ORGANIZATION For both source-domain pre-training and target-domain fine-tuning, each training mini-batch contains 64 person images of 16 actual or pseudo identities (4 for each identity). Note that the generated hard pseudo labels for the target-domain fine-tuning are updated after each epoch, so the mini-batch of target-domain images needs to be re-organized with updated hard pseudo labels after each epoch. All images are resized to 256 × 128 before being fed into the networks. All the hyper-parameters of the proposed MMT framework are chosen based on a validation set of the Duke-to-Market task with M t = 500 pseudo identities and IBN-ResNet-50 backbone. The same hyper-parameters are then directly applied to the other three domain adaptation tasks. We propose a two-stage training scheme, where ADAM optimizer is adopted to optimize the networks with a weight decay of 0.0005. Randomly erasing (b) is only adopted in target-domain fine-tuning. Stage 1: Source-domain pre-training. We adopt ResNet-50 or IBN-ResNet-50 as the backbone networks, where IBN-ResNet-50 achieves better performances by integrating both IN and BN modules. Two same networks are initialized with ImageNet pre-trained weights. Given the mini-batch of images, network parameters θ 1, θ 2 are updated independently by optimizing equation 3 with λ s = 1. The initial learning rate is set to 0.00035 and is decreased to 1/10 of its previous value on the 40th and 70th epoch in the total 80 epochs. Stage 2: End-to-end training with MMT. Based on pre-trained weights θ 1 and θ 2, the two networks are collaboratively updated by optimizing equation 9 with the loss weights λ t id = 0.5, λ t tri = 0.8. The temporal ensemble momentum α in equation 4 is set to 0.999. The learning rate is fixed to 0.00035 for overall 40 training epochs. We utilize k-means clustering algorithm and the number M t of pseudo classes is set as 500, 700, 900 for Market-1501 and DukeMTMC-reID, and 500, 1000, 1500, 2000 for MSMT17. Note that actual identity numbers in the target-domain training Table 1: Experimental of the proposed MMT and state-of-the-art methods on Market-1501 , DukeMTMC-reID , and MSMT17 datasets, where MMT-M t represents the with M t pseudo classes. Note that none of M t values equals the actual number of identities but our method still outperforms all state-of-the-arts. sets are different from M t. We test different M t values that are either smaller or greater than actual numbers. We compare our proposed MMT framework with state-of-the-art methods on the four domain adaptation tasks, Market-to-Duke, Duke-to-Market, Market-to-MSMT and Duke-to-MSMT. The are shown in Table 1. Our MMT framework significantly outperforms all existing approaches with both ResNet-50 and IBN-ResNet-50 backbones, which verifies the effectiveness of our method. Moreover, we almost approach fully-supervised learning performances without any manual annotations on the target domain. No post-processing technique, e.g. re-ranking (a) or multi-query fusion , is adopted. Specifically, by adopting the ResNet-50 backbone, we surpass the state-of-theart clustering-based SSG by considerable margins of 11.7% and 12.9% mAP on Market-to-Duke and Duke-to-Market tasks with simpler network architectures and lower output feature dimensions. Furthermore, evident 9.7% and 10.2% mAP gains are achieved on Market-to-MSMT and Duke-to-MSMT tasks. Recall that M t is the number of clusters or number of hard pseudo labels manually specified. More importantly, we achieve state-of-the-art performances on all tested target datasets with different M t, which are either fewer or more than the actual number of identities in the training set of the target domain. Such prove the necessity and effectiveness of our proposed pseudo label refinery for hard pseudo labels with inevitable noises. Table 2: Ablation studies of our proposed MMT on Duke-to-Market and Market-to-Duke tasks with M t of 500. Note that the actual numbers of identities are not equal to 500 for both datasets but our MMT method still shows significant improvements. To compare with relevant methods for tackling general noisy label problems, we implement Coteaching on unsupervised person re-ID task with 500 pseudo identities on the target domain, where the noisy labels are generated by the same clustering algorithm as our MMT framework. The hard classification (cross-entropy) loss is adopted on selected clean batches. All the hyper-parameters are set as the same for fair comparison, and the experimental are denoted as "Co-teaching -500" with both ResNet-50 and IBN-ResNet-50 backbones in Table 1. Comparing "Co-teaching -500 (ResNet-50)" with "Proposed MMT-500 (ResNet-50)", we observe significant 7.4% and 6.1% mAP drops on Market-to-Duke and Duketo-Market tasks respectively, since Co-teaching is designed for general close-set recognition problems with manually generated label noise, which could not tackle the real-world challenges in unsupervised person re-ID. More importantly, it does not explore how to mitigate the label noise for the triplet loss as our method does. In this section, we evaluate each component in our proposed framework by conducting ablation studies on Duke-to-Market and Market-to-Duke tasks with both ResNet-50 and IBN-ResNet-50 backbones. Results are shown in Table 2. Effectiveness of the soft pseudo label refinery. To investigate the necessity of handling noisy pseudo labels in clustering-based UDA methods, we create baseline models that utilize only off-line refined hard pseudo labels, i.e., optimizing equation 9 with λ t id = λ t tri = 0 for the two-step training strategy in Section 3.1. The baseline model performances are present in Table 2 as "Baseline (only L t id & L t tri)". Considerable drops of 17.7% and 14.9% mAP are observed on ResNet-50 for Duketo-Market and Market-to-Duke tasks. Similarly, 13.8% and 10.7% mAP decreases are shown on the IBN-ResNet-50 backbone. Stable increases achieved by the proposed on-line refined soft pseudo labels on different datasets and backbones demonstrate the necessity of soft pseudo label refinery and the effectiveness of our proposed MMT framework. Effectiveness of the soft softmax-triplet loss. We also verify the effectiveness of soft softmaxtriplet loss with softly refined triplet labels in our proposed MMT framework. Experiments of removing the soft softmax-triplet loss, i.e., λ Specifically, the mAP drops are 5.3% on ResNet-50 and 4.8% on IBN-ResNet-50 when evaluating on the target dataset Market-1501. As for the Market-to-Duke task, similar mAP drops of 3.6% and 4.0% on the two network structures can be observed. An evident improvement of up to 5.3% mAP demonstrates the usefulness of our proposed soft softmax-triplet loss. Effectiveness of Mutual Mean-Teaching. We propose to generate on-line refined soft pseudo labels for one network with the predictions of the past average model of the other network in our MMT framework, i.e., the soft labels for network 1 are output from the average model of network 2 and vice versa. We observe that the soft labels generated in such manner are more reliable due to the better decoupling between the past temporally average models of the two networks. Such a framework could effectively avoid bias amplification even when the networks have much erroneous outputs in the early training epochs. There are two possible simplification our MMT framework with less de-coupled structures. The first one is to keep only one network in our framework and use its past temporal average model to generate soft pseudo labels for training itself. Such experiments are denoted as "Baseline+MMT-500 (w/o θ 2)". The second simplification is to naïvely use one network's current-iteration predictions as the soft pseudo labels for training the other network and vice versa, i.e., α = 0 for equation 4. This set of experiments are denoted as "Baseline+MMT-500 (w/o E[θ])". Significant mAP drops compared to our proposed MMT could be observed in the two sets of experiments, especially when using the ResNet-50 backbone, e.g. the mAP drops by 8.9% on Duke-to-Market task when removing past average models. This validates the necessity of employing the proposed mutual mean-teaching scheme for providing more robust soft pseudo labels. In despite of the large margin of performance declines when removing either the peer network or the past average model, our proposed MMT outperforms the baseline model significantly, which further demonstrates the importance of adopting the proposed on-line refined soft pseudo labels. Necessity of hard pseudo labels in proposed MMT. Despite the robust soft pseudo labels bring significant improvements, the noisy hard pseudo labels are still essential to our proposed framework, since the hard classification loss L The initial network usually outputs uniform probabilities for each identity, which act as soft labels for soft classification loss, since it could not correctly distinguish between different identities on the target domain. Directly training with such smooth and noisy soft pseudo labels, the networks in our framework would soon collapse due to the large bias. One-hot hard labels for classification loss are critical for learning discriminative representations on the target domain. In contrast, the hard triplet loss L t tri is not absolutely necessary in our framework, as experiments without L t tri, denoted as "Baseline+MMT-500 (w/o L t tri)" with λ t tri = 1.0, show similar performances as our final with λ t tri = 0.8. It is much easier to learn to predict robust soft labels for the soft softmax-triplet loss in equation 8 even at early training epochs, which has only two classes, i.e., positive and negative. In this work, we propose an unsupervised Mutual Mean-Teaching (MMT) framework to tackle the problem of noisy pseudo labels in clustering-based unsupervised domain adaptation methods for person re-ID. The key is to conduct pseudo label refinery to better model inter-sample relations in the target domain by optimizing with the off-line refined hard pseudo labels and on-line refined soft pseudo labels in a collaborative training manner. Moreover, a novel soft softmax-triplet loss is proposed to support learning with softly refined triplet labels for optimal performances. Our method significantly outperforms all existing person re-ID methods on domain adaptation task with up to 18.2% improvements. Two temporal average models are introduced in our proposed MMT framework to provide more complementary soft labels and avoid training error amplification. Such average models are more de-coupled by ensembling the past parameters and provide more independent predictions, which is ignored by previous methods with peer-teaching strategy (; b). Despite we have verified the effectiveness of such design in Table 2 by removing the temporal average model, denoted as "Baseline+MMT-500 (w/o E[θ])", we would like to visualize the training process by plotting the KL divergence between peer networks' predictions for further comparison. As illustrated in Figure 3, the predictions by two temporal average models ("Proposed MMT-500") always keep a larger distance than predictions by two ordinary networks ("Proposed MMT-500 (w/o E[θ])"), which indicates that the temporal average models could prevent the two networks in our MMT from converging to each other soon under the collaborative training strategy. We utilize weighting factors of λ t tri = 0.8, λ t id = 0.5 in all our experiments by tuning on Duketo-Market task with IBN-ResNet-50 backbone and 500 pseudo identities. To further analyse the impact of different λ t tri and λ t id on different tasks, we conduct comparison experiments by varying the value of one parameter and keep the others fixed. Our MMT framework is robust and insensitive to different parameters except when the hard classification loss is eliminated with λ t id = 1.0. The weighting factor of hard and soft triplet losses λ t tri. In Figure 4 (a-b), we investigate the effect of the weighting factor λ t tri in equation 9, where the weight for soft softmax-triplet loss is λ t tri and the weight for hard triplet loss is (1 − λ t tri). We test our proposed MMT-500 with both ResNet-50 and IBN-ResNet-50 backbones when λ t tri is varying from 0.0, 0.3, 0.5, 0.8 and 1.0. Specifically, the soft softmax-triplet loss is removed from the final training objective (equation 9) when λ t tri is equal to 0.0, and the hard triplet loss is eliminated when λ t tri is set to 1.0. We observe

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A framework that conducts online refinement of pseudo labels with a novel soft softmax-triplet loss for unsupervised domain adaptation on person re-identification.

This paper proposes a deep generative classifier which is effective to detect out-of-distribution samples as well as classify in-distribution samples, by integrating the concept of Gaussian discriminant analysis into deep neural networks.