Question: Represent the article for finding a single-sentence TLDR

Question: Search engine has become a fundamental component in various web and mobile applications. Retrieving relevant documents from the massive datasets is challenging for a search engine system, especially when faced with verbose or tail queries. In this paper, we explore a vector space search framework for document retrieval. Specifically, we trained a deep semantic matching model so that each query and document can be encoded as a low dimensional embedding. Our model was trained based on BERT architecture. We deployed a fast k-nearest-neighbor index service for online serving. Both offline and online metrics demonstrate that our method improved retrieval performance and search quality considerably, particularly for tail queries Search engine has been widely applied in plenty of areas on the internet, which receives a query provided by users and returns a list of relevant documents within sub-seconds, helping users obtain their desired information instantaneously. Numerous technologies have been developed and utilized in real-world search engine systems. However, the existing semantic gap between search queries and documents, makes it challenging to retrieve the most relevant documents from tens of millions of documents. Therefore, there is still a large proportion of search requests that can not be satisfied perfectly, especially for long tail queries. A search engine system is usually composed of three main modules, -query understanding module -retrieval module -ranking module The query understanding module first parses the original query string into a structured query object BID32. More specifically, the query understanding module includes several subtasks, such as word segmentation, query correction, term importance analyze, query expansion, and query rewrite, etc. After the query string was parsed, an index module accepts the parsed query, and then retrieve the candidate documents. We call this stage the retrieval stage or the first round stage. Most web-scale search engine systems use the term inverted index for document retrieval, where term is the most basic unit in the whole retrieval procedure. In the first round stage, the retrieved documents are ranked by a simple relevance model, eg TF-IDF, BM25, and the top-N documents with the highest score are submitted to the next stage for ranking. Finally, the documents scored largest by a ranking function are returned to users eventually. For a search system described above, the final retrieval performance is highly enslaved by these query understanding module. Take word segmentation as an example: this task segments raw continuous query string into a list of segmented terms. Since the word segmentation algorithm has the risk of wrong segmentation. If the error segmented term does not appear in the document space, then no document could be retrieved in the first round stage, and it will return a page without any document which damages the user's experience seriously. There is a lot of work focused on better understanding queries to retrieve more relevant documents. However, since the final performance is influenced by all parts of the query understanding module. Attempts to optimize only one part is usually hard to contribute to a significant enhancement. To avoid the problems mentioned above, we propose a novel complementary retrieval sys-tem that retrieves documents without the traditional term-based retrieval framework. That is, instead of parse raw query into a structured query, we directly map both queries and documents into a low dimension of embedding. Then in the online serving, the k-nearest-neighbor documents of the given query in the latent embedding space are searched for retrieval. Recently, we have witnessed tremendous successful applications of deep learning techniques in information retrieval circle, like query document relevance matching BID14 BID34 BID33, query rewriting BID13, and search ranking BID12 BID10. However, it is still hard to directly retrieve relevant documents using an end2end fashion based on knearest-neighbor search in latent space, especially for long tail queries. The latest far-reaching advancement in natural language processing with deep learning, BERT BID8, provides a turning point to make end2end retrieval realizable. In this paper, we present a document retrieval framework as a supplement to the traditional inverted index based retrieval system. We design a new architecture to retrieve documents without a traditional term-based query understanding pipeline, which avoids performance decay by each subtask of query understanding. We use BERT architecture as the general encoder of query and document strings, then we fine-tuned the pre-trained BERT model with human annotated data and negative sampling technique. Finally, we conduct both offline and online experiments to verify our proposed method. To sum up, our main contributions are described below:1. We design a novel end2end document retrieval framework ，which is a supplement to traditional term-based methods.2. Our model is trained on transformer architecture, and a series of training techniques are developed for performance enhancement.3. The proposed techniques can not only be used in document retrieval but also have a significant improvement for search ranking. The rest of the paper is organized as follows. We concisely review the related work in Section 2. Sections 3 mainly describes our proposed methods. Offline and online experiments are detailed given in Section 4 and Section 5 respectively. Finally, we conclude and discuss future work in Section 6. There is a variety of work on search query understanding BID29, including query correction BID5, query term weighting BID39, query expansion and query reformulation BID3. In general, these kinds of methods coherently rewrite the raw query into a new query, by replacing, adding, or removing terms or phrases in the raw query. The rewritten query gets better expression and therefore can retrieve more relevant documents than the original one. Besides the inverted index, vector search engines BID9 have also been widely applied in many information seeking tasks, like image search BID16 and recommendation system BID6.To retrieve documents using a vector search, we need to map a piece of text into a low-dimensional numerical vector. Various embedding techniques have been developed and proven to have the powerful capability of capturing the semantic meaning of natural language text BID23, BID27 BID22. However, these kinds of models are still not capable of complicating text encoding, especially for long tail text queries. More recently, researchers have been describing the various architecture of neural models BID24. In text relevance matching area, we can divide most models into two typical categories, namely representation BID14 based models and interaction based models BID26 BID35 BID7. The representation models, like DSSM, are trained to obtain high-level representations of query and document respectively, then use vector distance between the query and document embedding for text relevance score. While the interaction based models first compute the term correlation matrix between query and doc-uments and calculate semantic matching similarity based on the correlation matrix. Both representation models and interaction based models could be trained from massive click feedback data BID18 BID0 or industrial annotation. These two kinds of model architecture are broadly deployed in realworld search engine systems, especially in ranking phase. For the representational models, once we obtained the high-level representation of raw texts, we can retrieve documents through the knearest-neighbor space search. However, the performance of representation based models are usually poorer than interaction based models, which makes k-nearest-neighbor retrieval hard to deploy in the real-world systems, since too many irrelevant documents retrieved may even damage overall performance. BID38 developed an architecture to transform the text into a sparse representation, while they still retrieve documents using a term-based index like lucene 1 because the nonzero value in the sparse representations is treated as virtual terms. BID2 BID11 ) developed a uniform query and document embedding framework by generating ngram embedding using user session and click data, and then generalize it to arbitrary text by mean average pooling of ngram embedding. Since ngram is a common and effective skill in a variety of NLP tasks, training a good ngram representation requires a massive of datasets, which may be a bottleneck for many researchers and companies. Meanwhile, the model capacity of DSSM and its' variations makes it not capable to capture complex semantic meanings of natural language. Recently, ELMo, GPT-2 BID31 and BERT BID8 show the great power of unsupervised pretraining in NLP tasks. The BERT model is built on a 12 layer transformer architecture, pre-trained with large scale text data. The pre-trained models can be fine-tuned easily and outperform many state-of-art models in various NLP tasks. We used the pre-trained BERT-Base(Chinese) 2 model released by Google and fine-tuned the model for semantic representation. Our fine-tuned model outperformed many state-of-art models in deep relevance matching, and obtain a great success in se-1 https://lucene.apache.org/ 2 https://github.com/google-research/bert In this section, we first illustrate our proposed semantic retrieval framework, which is composed of both offline and online parts respectively. Then, we introduce the model structure used for encoding queries and titles, and the techniques we used to boost the performance. Figure 1 shows our proposed system architecture. The offline module includes model training, document embedding inference, and semantic index builder. While in online serving, both query's semantic embedding and traditional term base query parser are computed, and then those two are sent to semantic index service and inverted index service respectively for document retrieval. Finally, documents retrieved from both index services are merged and sent to ranking service for document scoring. The pre-trained BERT model can be leveraged for semantic ranking and matching BID30 in various ways. We developed two models here: BERT(rep) and BERT(rel). The BERT(rep) model uses the pre-trained BERT model to obtain embedding of query and doc respectively, while the BERT(rel) model concatenates query and document first and get the one representation for a query document pair. The final score of a query, document pair is computed as below: DISPLAYFORM0 In the equation 1, we use the mean average of last layer as encoder output for each query and document, and compute the dot product of two embedding as matching score, where L represents the max sequence length. We also tried directly using the last layer of [CLS] term's embedding, but performed worse than the average pooling described in equation 1. DISPLAYFORM1 The equation 2 use embedding of last layer's [CLS] token and weighted sum it to a scalar by vector w, where w is a full connection layer with only one output. The model capacity of this method is more powerful than BERT(rel) because it calculates the term interaction between the query and document in the self-attention layers. However, since the BERT(rel) model is an interaction base model, this model can not be applied to semantic retrieval. Both two models are trained through a supervised learning fashion, with a pairwise max margin hinge loss to distinguish relatively positive and negative samples. The loss function for one query is: DISPLAYFORM2 where p i and p j represent to model score computed for each < query, document > pair, and y i and y j is the label for each document respectively. τ is the hyper parameter called margin to determine how far the model need to push a pair away from each other. The margin parameter is tuned for the best performance here. We use the additive data sampling technique to further enhance model performance. Therefore, the data we used to train our model is comprised of two parts, human annotated data, and negative sampled data. Negative sampling has been successfully applied in many tasks, such as neural language modelling BID23, e-commerce list embedding BID10, graph embedding BID37 and so on. Sampling negative training instance is also useful for model training in this scenario, since different from traditional term-based retrieval method, the vector space search is much more likely to retrieve irrelevant documents. Thus we propose to augment more irrelevant documents. When the negative samples were added to training, the model learned to push relevant and irrelevant documents away from each other, then the model is more robust to noisy documents. A straightforward way of negative sample mining is to select negative samples corresponding to a uniform distribution over the whole corpus, in particular, irrelevant documents here. However, this simple strategy fails to generate hard negative samples, which provide more important information for the model. Therefore, we propose another negative sampling method. At first, we train a baseline model with only human annotated data. Then we use this model to encode documents and queries. After that, we use an unsupervised cluster algorithm to assign each document and query a cluster id. Finally, we uniformly random selected negative documents from the cluster that query was distributed. For convenient, we call this kind of negative sampling name of N EG cluster, and globally sampled data name of N EG global. We append N EG cluster and N EG global to the raw dataset for per query and fine-tuned the model again to obtain our final model.reWe show the whole training procedure in the Algorithm 1 Training Framework of our proposed model human annotated data D, BERT pre-trained model M 1: M1 ← {D, M}, fine-tune the model M by D 2: compute embedding E for query and doc using M1 3: compute cluster centroids C by E 4: for all d ∈ Docs do 5:compute closest centroid C d for d 6: end for 7: for all q ∈ Query do 8:compute closest centroid Cq for q 9: uniform sample N EG global from whole doc set 10: uniform sample N EG cluster among docs where DISPLAYFORM0 D1(q) = {D(q) ∪ N EG global ∪ N EG cluster } 12: end for 13: M2 ← {D1, M}, fine-tune the model M by D1 Ensure:BERT model M2Algorithm 1, and sample's meaning is much closer to query than that of N EG global, which makes the model more robust for hard samples. Once the model was trained, we need to serve it on the fly. We first computed the embedding of all documents and build a vector index using faiss 3 BID19, which was open sourced by facebook and support k-nearestneighbor search for vector data in milliseconds. We developed a c++ based semantic index server to provide efficient concurrent online service. Our model was inferenced on a GPU server, and inference speed was accelerated 2 times faster than tfserving through a c++ based library developed by us. During the online serving, when a query was received, the GPU server first inferences the query embedding, and downstream sends the query embedding to semantic index service for document retrieval. For the balance of efficiency and effect, we retrieve k most similar documents in the semantic service for next stage ranking, where k is set to 20 here. In this section, we carry out offline experiments to illustrate the performance of our proposed semantic retrieval methods. In the experiment, we train the model with 1 epoch, use Adam with a learning rate of 10 −5, β1 = 0.9, β2 = 0.999. The data annotated by human editors is a list of triplets like <query, doc, relevance>. The rele-3 https://github.com/facebookresearch/faiss vance score has three grade 0, 1, 2, which represents bad, f air and excellent respectively. The dataset contains 36159 queries and 1181229 query doc pairs. Beside the dataset for training, we additionally annotated a small dataset for test, the test dataset contains 2703 queries and 84244 querydoc pairs. The summarize of dataset is shown at Table 2. We evaluate our proposed model from ranking and retrieval aspects. We compared the ranking performance using Normalized Discounted Cumulative Gain(NDCG), and retrieval performance with Recall. The way how these metrics are calculated will be introduced in Section 4.4 and Section 4.5 respectively. • ClickSim A relevance matching model BID17 which use web-scale click data to generate term representations for query and document, and use cosine similarity to represent query document relevance.• K-NRM An interaction based matching model using kernel pooling BID35.• Match Pyramid An interaction based matching model using convolutions on term matching matrix BID25 ).• DSSM A representation based model proposed by Microsoft Research BID14. The model proposed here using word vectors pretrained on document title corpus. And three full connection layer with size of 300, 300, and 128 dimensions are used for text encoding. We use metric Recall to evaluate the model's retrieval performance here. This metric measures how many relevant documents are retrieved by a given model. For a given query q, the Recall rate is calculated as, DISPLAYFORM0 where Ret q represents the retrieved documents for q, Rel q stands for all the relevant documents for query q, where relevant documents are defined as document relevance annotated larger than 0 here. To evaluate the recall performance offline, we first built semantic index both for our model and baseline model. We computed representation for document title of each model, then we used the representation embedding to build semantic index. Once queries' embedding of each model were computed, we retrieved the top k documents by knearest-neighbor search. Besides comparing the recall measure of different models only using semantic index, we compared the recall enhancement when the semantic index was added to the lexical inverted index. We used a commercial term-based inverted index engine developed by us and build a lexical index with it. Both lexical inverted index and semantic index were built to retrieve documents, with top 300 and top 20 respectively. Then we calculated the recall of the union set. In the experiment, since document size of testset is small, we need a larger document corpus to make the recall measured more accurately. Therefore, both semantic index and lexical index were built with all human annotated data, including trainset and testset. And recall metric were calculated using only queries in the testset. TAB3 shows the of different models, BERT(rep) outperforms baseline model DSSM significantly in the recall measure. And after adding our model as a supplement to the lexical index, the recall rate is improved from 54.9% to 69.4%. While the baseline model, DSSM performs poorly on this task. -NDCG score Since our proposed model could not only be applied in document retrieval but also applied in the ranking stage. We measured the model's ranking quality through Normalized Discounted Cumulative Gain(NDCG). For a ranked document list, the NDCG for a query is calculated as, DISPLAYFORM0 where IDCG n represents the DCG score when the list was perfectly ranked by relevance. We compute following variation of Discounted Cumulative Gain(DCG) BID15, DISPLAYFORM1 According to the equation 6, higher relevance label contribute to higher weight in the computation. We calculate N DCG with different rank list size of {1, 3, 5} respectively. Table 4 shows that our model is superior to the state of art deep relevance matching models, and BERT(rep) model is slightly worse than the BERT(rel) model since BERT(rel) model uses self-attention between the query and title tokens before aggregates final score. However, both the BERT(rep) model and BERT(rel) model outperform other baselines significantly. We feed the doc product of query doc embedding into a gbdt ranking model BID4 as a relevance feature, and observe the feature importance after the tree model was trained. The feature importance was computed by the statistics collected during the tree ensemble training procedure. TAB5 shows that without adding BERT(rel) feature, the BERT(rep) feature ranks first in the ranking function, and accounts for 34% of importance among all features in our ranking function. In Section 3.2.1, we described two negative sampling generator method: the N EG global samples and N EG cluster for training data enhancement. We tuned the negative samples size, and obtained the best performance with 10 N EG global and 10 N EG cluster respectively. After adding negative samples, the average negative sample size for a given query increased from 19.9 to 39.9. TAB6 shows the model performance with different kinds of negative samples. Only adding N EG global can improve NDCG@3 at about 0.5%, when adding N EG cluster, the NDCG@3 is further improved by 0.8%. Therefore, the overall measurements are enhanced by 1.4% after additive sampling. pooling layer used in BERT model In this paper, we use the reduce-mean of the last layer as BERT(rep) model's pooled output. Different layers of BERT may own different aspects of knowledge about the input sequence. To verify the effectiveness of different layers, we trained different models, with pooled output from different layers respectively. From FIG1, the red solid line shows that the layer closest to last obtains higher NDCG measure. This is reasonable since higher layers make the model contains more parameters. Besides comparing the of different layers, we also developed a method aggregate the embedding of all layers. In this method, an attention layer calculates the weight across different layers, therefore a weighted sum of each layer's embedding on each position is the final representation of each term. After that, we used reduce-mean of all terms' embedding as the final pooled output. The of aggregation is shown as the green dot line, which does not outperform simple average pooling on the last layer. Meanwhile, we also tried using [CLS] term's embedding of the last layer as pooled output, but it behaved even worse. In , using mean average pooling of the last layer as final pooled output performs best in this scenario, even though some work claims aggregating layers is useful BID21.5 Online Evaluation and Case Study After offline evaluations, we conduct an online a/b test to further verify our proposed system. In the online experiment procedure, 40 percent of online traffic were randomly distributed to four groups, 2 control groups, and 2 experimental groups. The metric we used to evaluate is the Clicked Search Rate(CSR), which is computed as: DISPLAYFORM0 After a week's observation, as shown in TAB7, the overall CSR of two experimental groups both surpass two control groups by 0.65%, which is relatively a huge improvement to our experience. We also examined the online performance for queries with different frequency. We split queries into Top, Torso, and Tail by query search times in a day. Since our proposed method mainly focuses on boosting the performance of long tail queries, we can see the CSR metric is not significant in the Top and Torso query part. But the metric increased by nearly 1.05 % in the Tail part, which contributed to the largest algorithm iteration in the first half of 2019. This section highlights some good cases after our system was deployed online. We show the final ranked at top 6 for query "送外卖不认识路" (do not know the way to deliver food) at TAB8, where SEMANTIC represents the document retrieved from the proposed semantic index, and LEXICAL for traditional termbased inverted index. In this case, three documents are retrieved from semantic index, and the relevance is also much better than the document from traditional inverted index. Notice that there are many ways to express "不 认 识 路"(do not know the way) in Chinese, while the semantic index retrieved documents indeed capture the several alternatives of expressing it: "不知道路线", "不认路", "不懂路". And the term retrieved document only contains the same term "不认识路" as query expressed. In this paper, we present an architecture for semantic document retrieval. In this architecture, we first train a deep representation model for query and document embedding, then we build our semantic index using a fast k-nearest-neighbor vector search engine. Both offline and online experiments have shown that retrieval performance is greatly enhanced by our method. For the future work, we would like to explore a more general framework that could use more signals involved for semantic retrievals, like document quality features, recency features, and other text encoding models.

Represent the TLDR for finding the corresponding scientific article

A deep semantic framework for textual search engine document retrieval

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Semi-Supervised Learning (SSL) approaches have been an influential framework for the usage of unlabeled data when there is not a sufficient amount of labeled data available over the course of training. SSL methods based on Convolutional Neural Networks (CNNs) have recently provided successful on standard benchmark tasks such as image classification. In this work, we consider the general setting of SSL problem where the labeled and unlabeled data come from the same underlying probability distribution. We propose a new approach that adopts an Optimal Transport (OT) technique serving as a metric of similarity between discrete empirical probability measures to provide pseudo-labels for the unlabeled data, which can then be used in conjunction with the initial labeled data to train the CNN model in an SSL manner. We have evaluated and compared our proposed method with state-of-the-art SSL algorithms on standard datasets to demonstrate the superiority and effectiveness of our SSL algorithm. Recent developments in CNNs have provided promising for many applications in machine learning and computer vision;. However, the success of CNN models requires a vast amount of well-annotated training data, which is not always feasible to perform manually. There are essentially two different solutions that are usually used to deal with this problem: 1) Transfer Learning (TL) and 2) SemiSupervised Learning (SSL). In TL methods , the learning of a new task is improved by transferring knowledge from a related task which has already been learned. SSL methods , however, tend to learn discriminative models that can make use of the information from an input distribution that is given by a large amount of unlabeled data. To make use of unlabeled data, it is presumed that the underlying distribution of data has some structure. SSL algorithms make use of at least one of the following structural assumptions: continuity, cluster, or manifold. In the continuity assumption, data which are close to each other are more likely to belong to the same class. In the cluster assumption, data tends to form discrete clusters, and data in the same cluster are more likely to share the same label. In the manifold assumption, data lies approximately on a manifold of much lower dimension than the input space which can be classified by using distances and densities defined on the manifold. Thus, to define a natural similarity distance or divergence between probability measures on a manifold, it is important to consider the geometrical structures of the metric space in which the manifold exists. There are two principal directions that model geometrical structures underlying the manifold on which the discrete probability measures lie. The first direction is based on the principal of invariance, which relies on the criterion that the geometry between probability measures should be invariant under invertible transformations of random variables. This perspective is the foundation of the theory of information geometry, which operates as a base for the statistical inference. The second direction is established by the theory of Optimal Transport (OT), which exploits prior geometric knowledge on the base space in which random variables are valued. Computing OT or Wasserstein distance between two random variables equals to achieving a coupling between these two variables that is optimal in the sense that the expectation of the transportation cost between the first and second variables is minimal. The Wasserstein distance between two probability measures considers the metric properties of the base space on which a structure or a pattern is defined. However, traditional information-theoretic divergences such as the Hellinger divergence and the Kullback-Leibler (KL) divergence are not able to properly capture the geometry of the base space. Thus, the Wasserstein distance is useful for the applications where the structure or geometry of the base space plays a significant role. In this work, similar to other SSL methods, we make a structural assumption about the data in which the data are represented by a CNN model. Inspired by the Wasserstein distance, which exploits properly the geometry of the base space to provide a natural notion of similarity between the discrete empirical measures, we use it to provide pseudo-labels for the unlabeled data to train a CNN model in an SSL fashion. Specifically, in our SSL method, labeled data belonging to each class is a discrete measure. Thus, all the labeled data create a measure of measures and similarly, the pool of unlabeled data is also a measure of measures constructed by data belonging to different classes. Thus, we design a measure of measures OT plan serving as a similarity metric between discrete empirical measures to map the unlabeled measures to the labeled measures based on which, the pseudo-labels for the unlabeled data are inferred. Our SSL method is based on the role of Wasserstein distances in the hierarchical modeling. It stems from the fact that the labeled and unlabeled datasets hierarchically create a measure of measures in which each measure is constructed by the data belonging to the same class. Computing the exact Wasserstein distance, however, is computationally expensive and usually is solved by a linear program (Appendix A and D). introduced an interesting method which relaxes the OT problem using the entropy of the solution as a strong convex regularizer. The entropic regularization provides two main advantageous: 1) The regularized OT problem relies on Sinkhorns algorithm that is faster by several orders of magnitude than the exact solution of the linear program. 2) In contrast to exact OT, the regularized OT is a differentiable function of their inputs, even when the OT problem is used for discrete measures. These advantages have caused that the regularized OT to receive a lot of attention in machine learning applications such as generating data; , designing loss function , domain adaptation; , clustering; and low-rank approximation. Pseudo-Labeling is a simple approach whereby a model incorporates it's own predictions on unlabeled data to obtain additional information during the training;;. The main downside of these methods is that they are unable to correct their own mistakes where predictions of the model on unlabeled data are confident but incorrect. In such a case, the erroneous data not only can not contribute to the training, but the error of the models is amplified during the training as well. This effect is aggravated where the domain of the unlabeled data is different from that of labeled data. Note that pseudo-labeling in is similar to entropy regularization , in the sense that it forces the model to provide higher confidence predictions for unlabeled data. However, it differs because it only forces these criteria on data which have a low entropy prediction due to the threshold of confidence. Consistency Regularization can be considered as a way of using unlabeled data to explore a smooth manifold on which all of the data points are embedded. This simple criterion has provided a set of methods that are currently considered as state of the art for the SSL challenge. Some of these methods are stochastic perturbations Sajjadi et al. (2016b), π-model , mean teacher , and Virtual Adversarial Training (VAT). The original idea behind stochastic perturbations and π-model was first introduced in and has been referred to as pseudo-ensembles. The pseudo-ensembles regularization techniques are usually designed such that the prediction of the model ideally should not change significantly if the data given to the model is perturbed; in other words, under realistic perturbations of a data point x (x → x), output of the model f θ (x) should not change significantly. This goal is achieved by adding a weighted loss term such as d(f θ (x), f θ (x)) to the total loss of the model f θ (x), where d(., .) is mean squared error or Kullback-Leibler divergence which measures a distance between outputs of the prediction function. The main problem of pseudo-ensemble methods, including π-model is that they rely on a potentially unstable target prediction, which can immediately change during the training. To address this problem, two methods, including temporal ensembling and mean teacher , were proposed to obtain a more stable target output f θ (x). Specifically, temporal ensembling uses an exponentially accumulated average of outputs, f θ (x), to make the target output smooth and consistent. Inspired by this method, mean teacher instead uses a prediction function which is parametrized by an exponentially accumulated average of θ during the training. Like the π-model, mean teacher adds a mean squared error loss d(f θ (x), f θ (x)) as a regularization term to the total loss function for training the network. It has been shown that mean teacher outperforms temporal ensembling in practice. Contrary to stochastic perturbation methods which rely on constructing f θ (x) stochastically, VAT in the first step approximates a small perturbation r to add it to x which significantly changes the prediction of the model f θ (x). In the next step, a consistency regularization technique is applied to minimize d(f θ (x), f θ (x + r)) with respect to θ which is the parameters of the model. Entropy Minimization methods use a loss term which is applied on the unlabeled data to force the model f θ (x) to produce confident predictions (i.e., low-entropy) for all of the samples, regardless of what the actual labels are. For example, by assuming the softmax layer of a CNN has c outputs, the loss term applied on unlabeled data is as follows: Ideally, this class of methods penalizes the decision boundary that passes near the data points, while they instead force the model to provide a high-confidence prediction. It has been shown that entropy minimization on its own, can not produce competitive Sajjadi et al. (2016a). However, entropy minimization can be used in conjunction with VAT (i.e., EntMin VAT) to provide state of the art in which VAT assumes a fixed virtual label prediction in the regularization d(f θ (x), f θ (x + r)). For any subset θ ⊂ R c, assume that S(θ) represents the space of Borel probability measures on θ. The Wasserstein space of order k ∈ [1, ∞) of probability measures on θ is defined as follows: where, ||.|| is the Euclidean distance in R c. Let Π(P, Q) denote the set of all probability measures on θ × θ which have marginals P and Q; then the k-th Wasserstein distance between P and Q in S k (θ), is defined as follows: where x ∼ P, x ∼ Q and k ≥ 1. Explicitly, W k (P, Q) is the optimal cost of moving mass from P to Q, where the cost of moving mass is proportional to the Euclidean distance raised to the power k. In Eq., the Wasserstein between two probability measures was defined. However, using a recursion of concepts, we can talk about measure of measures in which a cloud of measures (M) is transported to another cloud of measures (M). We define a relevant distance metric on this abstract space as follows: let the space of Borel measures on S k (θ) be represented by S k (S k (θ)); this space is also a Polish, complete and separable metric space as S k (θ) is a Polish space (cf. section. 3 in). It will be endowed with a Wasserstein metric W k of order k that is induced by a metric where, Q ∼ M, P ∼ M, and Π(M, M) is the set of all probability measures on S k (θ) × S k (θ) that have marginals M and M. Note that the existence of an optimal solution, π ∈ Π(M, M), is always guaranteed (Appendix E). In words, W k (M, M) corresponds to the optimal cost of transporting mass from M to M, where the cost of moving unit mass in its space of support, S k (θ), is proportional to the power k of the Wasserstein distance W k in S k (θ). The goal of our algorithm is to use OT to provide pseudo-labels for the unlabeled data to train a CNN model in an SSL manner. The basic premise in our algorithm is that the discrepancy between two discrete empirical measures which come from the same underlying distribution is expected to be less than the case where these measures come from two different distributions. In this work, since we make a structural assumption about the data and assume that the labeled and unlabeled data belonging to the same class come from the same distribution (i.e., general setting in SSL), we leverage OT metric to map similar measures from two measure of measures. This is because OT exploits well the structure or geometry of the underlying metric space to provide a natural notion of similarity between empirical measures in the metric space. Here, labeled data belonging to the same class is a measure. Thus, all the initially labeled data construct a measure of measures and similarly, all the unlabeled data is also a measure of measures constructed by data from different classes. Thus, we design a measure of measures OT plan to map the unlabeled measures to the similar labeled measures based on which, pseudo-labels for the unlabeled data in each measure are inferred. The mapping between the labeled and unlabeled measures based on the measure of measures OT is formulated as follows: Given an image z i ∈ R m×n from the either labeled or unlabeled dataset, the CNN acts as a function f (w, z i): R m×n → R c with the parameters w that maps z i to a c-dimensional representation, where c is number of the classes. Assume that X = {x 1, ..., x m} and X = {x 1, ..., x m} are the sets of c-dimensional outputs represented by the CNN for the labeled and unlabeled images, respectively. Let P i = 1/n i ni j=1 δ xj denote a discrete measure constructed by the labeled data belonging to the i-th class, where δ xj is a Dirac unit mass on x j and n i is number of the data within the i-th class. Thus, all the labeled data construct a measure of measures M = c i=1 α i δ Pi, where α i = n i /m represents amount of the mass in the measure P i and δ Pi is a Dirac unit mass on the measure P i. Similarly unlabeled data construct a measure of measures M = c j=1 β j δ Qj in that each measure Q i, is created by the unlabeled data belonging to the unknown but the same class, where β j = n j /m is amount of the mass in the measure Q j and δ Qi is a Dirac unit mass on Q j. The goal of our SSL method is to use the OT to find a coupling between the measures in M and M that is optimal in the sense that it has a minimal expected transportation cost. This is because the transportation cost between two empirical measures which come from the same distribution (data from the same class) is expected to be less than the case where these measures come from two different distributions (data from different classes). Thus, we design an OT cost function defined in Eq. to obtain an optimal coupling between measures in M and M based on which the labels of data in the unlabeled measures are inferred: where T is the optimal coupling matrix in which T (i, j) indicates amount of the mass that should be moved from Q i to P j to provide an OT plan between M and M. Thus, if highest amount of the mass from Q i is transported to P k (i.e., Q i is mapped to P k); the data belonging to the measure Q i are annotated by k which is the label of the measure P k. Variable X is the pairwise similarity matrix between measures within M and M in which X(i, j) = W k (Q i, P j) which is the Wasserstein distance between two clouds of data points Q i and P j. Note that the ground metric used for computing W k (Q i, P j) is the Euclidean distance. Moreover, T, M denotes the Frobenius dot-product between T and X matrices, and T is transportation polytope defined as follows: where 1 c is a c-dimensional vector with all elements equal to one. Finally, E(T) is entropy of the optimal coupling matrix T which is used for regularizing the OT, and λ is a hyperparameter that balances between two terms in Eq.. The optimal coupling solution for the regularized OT defined in Eq. is obtained by an iterative algorithm relied on Sinkhorn algorithm (Appendix D). In Sec. 4, we represented the pool of unlabeled data as a measure of measures M = c j=1 β j δ Qj in which each measure is constructed by data that belong to the same class. However, label of the unlabeled data is unknown to allow us to identify these unlabeled measures. Moreover, CNN as a classifier trained on a limited amount of the labeled data simply miss-classifies these unlabeled data. In such a case, there is little option other than to use unsupervised methods, such as the clustering to explore the unlabeled data belonging to the same class. This is because in structural assumption based on the clustering, it is assumed that the data within the same cluster are more likely to share the same label. Here, we leverage the Wasserstein metric to explore these unknown measures underlying the unlabeled data. Specifically, we relate the clustering algorithm to the problem of exploring Wasserstein barycenter of the unlabeled data. Wasserstein barycenter was initially introduced by. Given probability measures R 1,..., R l ∈ S 2 (θ) for l ≥ 1, their Wasserstein barycenterR l,µ is defined as follows: where µ i is the weight associated with R i. In the case where R 1,..., R l are discrete measures with finite number of elements and the weights in µ are uniform, it is shown by that the problem of exploring Wasserstein barycenterR l,µ on the space of S 2 (θ) in is recast to search only on O r (θ) denoting as a set of probability measures with at most r support points in θ, where r = l i=1 e i − l + 1 and e i is the number of elements in R i for all 1 ≤ i ≤ l. Moreover, an efficient algorithm for exploring local solutions of the Wasserstein barycenter problem over O r (θ) for some r ≥ 1 has been studied by. Beside, the popular K-means clustering can be considered as solving an optimization problem that comes up in the quantization problem, a simple but very practical connection;. The connection is as follows: Given m unlabeled data x 1,..., x m ∈ θ. Suppose that these data are related to at most k clusters where k ≥ 1 is a given number. The K-means problem finds the set Z containing at most k atoms θ 1,..., θ k ∈ θ that minimizes: is equivalent to explore a discrete measure H including finite number of support points and minimizing the following objective:. This problem can also be thought of as a Wasserstein barycenter problem when l = 1. From this prospective, as denoted by , the algorithm for finding the Wasserstein barycenters is an alternative for the popular Loyds algorithm to find local minimum of the K-means objective. Thus, we adopt the algorithm introduced in used for computing the Wasserstein barycenters of empirical probability measures to explore the clusters underlying the unlabeled data (Appendix B). Our SSL method finally leverages the unlabeled image data annotated by pseudo-labels obtained from the OT in conjunction with the supervision signals of the initial labeled image data to train the CNN classifier. Thus, we use the generic cross entropy as our discriminative loss function to train the parameters of our CNN as follows: Let X l be all of the labeled training data annotated by true labels Y, and X u be the unlabeled training data annotated by pseudo-labels Y, then the total loss function L, used to train our CNN in an SSL fashion is as follows: where w is parameters of the CNN, and L c denotes cross entropy loss function, and α is a hyperparameter that balances between two losses obtained from the labeled and unlabeled data. For training, we initially train the CNN using the labeled data as a warm up step, and then use OT to provide pseudo-labels for the unlabeled data to train the CNN in conjunction with the initial labeled data for the next epochs. Specifically, after training the CNN using the labeled data, in each epoch, we select the same amount of initial labeled data from the pool of unlabeled data and then use OT to compute their pseudo-labels; then, we train the CNN in a mini-batch mode. Our overall SSL method is described in Algorithm 2 (Appendix C). For evaluating our SSL technique and comparing it with the other SSL algorithms, we follow the concrete suggestions and criteria which are provided in. Some of these recommendations are as follows: 1) we use a common CNN architecture and training procedure to conduct a comparative analysis, because differences in CNN architecture or even implementation details can influence the . 2) We report the performance of a fully-supervised case as a baseline because the goal of SSL is to greatly outperform the fully-supervised settings. 3) We change the amount of labeled and unlabeled data when reporting the performance of our SSL algorithm because an ideal SSL method should remain efficient even with the small amount of labeled and additional unlabeled data. 4) We also perform an analysis on realistic small validation sets. This is because, in real-world applications, the large validation set is instead used as the training, therefore, an SSL algorithm which needs heavy tuning on a per-task or per-model basis to perform well would not be applicable if the validation sets are realistically small (This analysis is done in Appendix F). For the first criterion, we have used the'WRN-28-2' model (i.e., ResNet with depth 28 and width 2) , including batch normalization and leaky ReLU nonlinearities. We conducted our experiments on the widely used CIFAR-10 , and datasets. Note that in our experiments, we tackle the general SSL challenge where the labeled and unlabeled data come from the same underlying distribution, and a given unlabeled data belongs to one of the classes in the labeled set and therefor, there is no class distribution mismatch. Moreover, for each of these datasets, we split the training set into two different sets of labeled and unlabeled data. For training, we use the well-known Adam optimizer with the default hyperparameters values and a learning rate of 3 × 10 −3 in our experiments, and all the experiments have been done on a NVIDIA TITAN X GPU. The batch size in our experiments is set to 100. We have not used any form of early stopping; however, we have consistently monitored the performance of the validation set and reported test error at the point of lowest validation error. The stopping criteria for the Sinkhorn algorithm is either maxIter = 10,000 or tolerance = 10 −8, where maxIter is the maximum number of iterations and tolerance is a threshold for the integrated stopping criterion based on the marginal differences. In experiments, we followed the data augmentation and standard data normalization used in. Specifically, for SVHN, we converted pixel intensity values of the images to floating point values in the range of [-1, 1]. For the data augmentation, we only applied random translation by up to 2 pixels. We used the standard training and validation split, with 65,932 images for the training set and 7,325 for the validation set. For CIFAR-10, we applied global contrast normalization. The data augmentation on CIFAR-10 are random translation by up to 2 pixels, random horizontal flipping, and Gaussian input noise with standard deviation 0.15. We used the standard training and validation split, with 45,000 images for the training set and 5,000 images for the validation set. Here, we consider the second criterion for evaluation of our SSL method. The purpose of SSL is mainly to achieve a better performance when it uses the unlabeled data than the case where using the labeled data alone. To ensure that our SSL model benefits from the unlabeled data during the training, we report the error rate of the WRN model for both cases where we only use the labeled data (i.e., Supervised in Table. 1), and the case where we leverage the unlabeled data by using the OT technique during the training (i.e., ROT in Table. 1). Moreover, we have reported the performance of other SSL algorithms in Table. 1 which also leverage the unlabeled data during the training. All of the compared SSL methods use the common CNN model (i.e., 'WRN-28-2') and training procedure as suggested in the first criterion for the realistic evaluation of SSL models. The of all SSL methods reported in Table. 1 is the test error at the point of lowest validation error for tuning their hyperparameters. For a fair evaluation with other SSL algorithms, we selected 4,000 samples of the training set as the labeled data and the remaining as the unlabeled data for the CIFAR-10 dataset, and we chose 1,000 samples of the training set as the labeled data and the rest as the unlabeled data for the SVHN dataset. We ran our SSL algorithm over five times with different random splits of labeled and unlabeled sets for each dataset, and we reported the mean and standard deviation of the test error rate in Table. 1. The in Table. 1 indicates that on both CIFAR-10 and SVHN, the gap between the fully-supervised baseline and ROT is bigger than this gap for the other SSL methods. This indicates the potential of our model for leveraging the unlabeled data in comparison to other methods that also use the unlabeled data to improve the classification performance of a CNN model in SSL fashion. Moreover, we trained our baseline WRN on the entire training set of CIFAR-10 and SVHN and the test error over five runs are 4.23(±0.18) and 2.56(±0.04), respectively. Besides the particular manner in which we choose the one particular pseudo-label, we also use "soft pseudo-labels". Essentially, instead of having the one-hot target in the usual classification loss (i.e., cross-entropy), we can have the row of the transport plan corresponding to the unlabeled data points as the target. We used the soft pseudo-labels produced by OT to train the CNN. The comparison of in Table. 1 show that one-hot targets used in ROT outperforms the soft pseudo-labels used in ROT. Why this is happening can be supported by SSL methods based on the entropy minimization criterion. This set of methods force the model to produce confident predictions (i.e., low entropy for output of the model). Similarly here, once we use one-hot targets, we encourage the network to produce more confident predictions than when using soft-pseudo labels. In this section, we compare ROT which is based on the measure of measure OT with two other baselines. Both the baselines assign pseudo-labels for the unlabeled samples based on the greedy nearest neighbor (GNN) search. The first baseline is sample to sample (S-S-GNN) case, where pseudolabels for the unlabeled data are obtained by GNN on the outputs of softmax layer. Specifically, for each of the unlabeled sample, we annotate it with the label of the closest labeled sample in the training set. The second baseline is sample to measure (S-M-GNN) case where, pseudo-labels of the unlabeled samples are obtained based on the GNN between the unlabeled samples and the probability measures constructed by initial labeled data in the training set. When transporting from a Dirac to a probability measure, the OT problem (regularized or not) has a closed form. Essentially, there is only one admissible coupling. Thus, in such a case, the Wasserstein distance between a sample to a probability measure is simply computed as follows: Given an unlabeled Dirac δ x i and a labeled measure The comparison of between ROT, and these baselines on the SVHN and CIFAR-10 in Table. 2 shows the benefit of measure of measure OT for training a CNN in an SSL manner. Instead of using the CNN as a classifier to produce pseudo-labels for the unlabeled data, we used the Wasserstein barycenters to cluster the unlabeled data. This allowed us to explore the unlabeled measures that we could then match them with the labeled measures for pseudo-labeling. This was because the CNN, as a classifier trained on a limited amount of the labeled data, simply miss-classifies the unlabeled data. To compare these two different strategies for producing the pseudo-labels to train the CNN classifier in an SSL fashion, we experimentally show how the clustering-based method (i.e., ROT) can have a greater positive influence on the training of our CNN classifier. We report the number of pseudo-labels which are accurately predicted by ROT. This allows us to know the level of accuracy of the pseudo-label obtained for the unlabeled data, which the CNN can then benefit from during the training. We also report these with that of predicted labels achieved by the baseline CNN classifier (i.e., WRN) on the unlabeled training data. This comparison also allows us to know whether or not the CNN classifier can benefit from our strategy for providing pseudo-labels during the training, because, otherwise, the WRN can simply use its own predicted labels on unlabeled training data over the course of training. To indicate the efficiency of our method during the training of the CNN, we changed the number of initial labeled data in the training set and reported the number of accurately predicted pseudo-labels by the baseline WRN, and ROT on the remaining unlabeled training data. Fig. 2 (c) and Fig. 1(d) show that, for both CIFAR and SVHN datasets, the labels predicted by ROT on the unlabeled training data are more accurate than the WRN, which means that the entire CNN network can better benefit from the ROT strategy than the case where it is trained solely by its own predicted labels. Moreover, we monitored the trend of transportation cost between the labeled and unlabeled measures obtained by Eq. 3 during the training. Fig. 2(a) and Fig. 2(b) show that the transportation cost is reduced as the images fed into the CNN are represented by a better feature set during the training. In Table. 2, we evaluated ROT for the case where we only use 4,000 and 1,000 initial labeled data for the CIFAR-10 and SVHN, respectively. However, here, we explore that how varying the amount of initial labeled data decreases the performance of ROT in the very limited label regime, and also at which point our SSL method can recover the performance of training when using all of the labeled data in the dataset. To do this evaluation, we gradually increase the number of labeled data during the training and report the performance of our SSL method on the testing set. In this experiment, we ran our SSL method over five times with different random splits of labeled and unlabeled sets for each dataset, and reported the mean and standard deviation of the error rate in Fig. 2(a) and Fig. 2(b). The show that the performance of ROT tends to converge as the number of labels increases. Another possibility for evaluating the performance of our SSL method is to change the number of unlabeled data during the training. However, using the CIFAR-10 and SVHN datasets in isolation puts an upper limit on the amount of available unlabeled data. Fortunately, in contrast to CIFAR-10, SVHN has been distributed with the SVHN-extra dataset, which includes 531,131 additional digit images and has also been previously used as unlabeled data for evaluation of different SSL methods in. These additional data come from the same distribution as SVHN does, which allows us to use them in our SSL framework. Fig. 2(c) shows the trend of test error for our SSL algorithm on SVHN with 1,000 labels and changing amounts of unlabeled images from SVHN-extra dataset. The shows that, increasing the amount of unlabeled data improves the performance of our SSL method, but this improvement is not significant when we provide 40k unlabeled data. We proposed a new SSL method based on the optimal transportation technique in which unlabeled data masses are transported to a set of labeled data masses, each of which is constructed by data belonging to the same class. In this method, we found a mapping between the labeled and unlabeled masses which was used to infer pseudo-labels for the unlabeled data so that we could use them to train our CNN model. Finally, we experimentally evaluated our SSL method to indicate its potential and effectiveness for leveraging the unlabeled data when labels are limited during the training. Discrete Optimal Transport: For any r ≥ 1, let the probability simplex be denoted by, and also assume that U = {u 1, ..., u n} and V = {v 1, ..., v m} are two sets of data points in between two discrete measures U and V is the k-th root of the optimum of a network flow problem known as the transportation problem. Note that δ ui is the Dirac unit mass located on point u i, a and b are the weighting vectors which belong to the probability simplex ∆ n and ∆ m, respectively. The transportation problem depends on the two following components: 1) matrix M ∈ R n×m + which encodes the geometry of the data points by measuring the pairwise distance between elements in U and V increased to the power k, 2) the transportation polytope P (a, b) ∈ R n×m + which acts as a feasible set, characterized as a set of n × m non-negative matrices such that their row and column marginals are a and b, respectively. This means that the transportation plan should satisfy the marginal constraints. In other words, let 1 m be an m-dimensional vector with all elements equal to one, then the transportation polytope is represented as follows: P (a, b) = {T ∈ R n×m + |T 1 n = b, T 1 m = a}. Essentially, each element T (i, j) indicates the amount of mass which is transported from i to j. Note that in the transportation problem, the matrix M is also considered as a cost parameter such that Let T, M denote the Frobenius dot-product between T and M matrices. Then the discrete Wasserstein distance W k (U, V) is formulated by an optimum of a parametric linear program g on a cost matrix M, and n × m number of variables parameterized by the marginals a and b as follows: The Wasserstein distance in is a Linear Program (LP) and a subgradient of its solution can be calculated using Lagrange duality. The dual LP of is formulated as follows: where the polyhedron C M of dual variables is as follows: Considering LP duality, the following equality is established. Computing the exact Wasserstein distance in is time consuming. To alleviate this problem, has introduced an interesting method that regularizes using the entropy of the solution matrix H(T), (i.e., min T, M + γH(T)). It has been shown that if T γ is the solution of the regularized version of and α γ is its dual solution in, then ∃!u ∈ R n +, v ∈ R m + such that the solution matrix is T γ = diag(u)Kdiag(v) and α γ = − log(u)/γ + (log(u) 1 n )/(γn))1 n where, K = exp(−M/γ). The vectors u and v are updated iteratively between step 1 and 2 by using the well-known Sinkhorn algorithm as follows: step 1)u = a/Kv and step 2)v = b/K u, where/ denotes element-wise division operator. Given an image x n ∈ R m×n from the either labeled or the unlabeled set, the CNN acts as a function f n: R m×n → R c with the parameters θ n that maps x n to a c-dimensional representation, where c is the number of classes. Assume that X u = {x 1, ..., x n} is the set of CNN outputs extracted from the unlabeled data. As noted in , the Wasserstein barycenter of the unlabeled set X u is equivalent to Lloyd's algorithm, where the maximization step (i.e., the assignment of the weight of each data point to its closest centroid) is equivalent to the computation of α in dual form, while the expectation step (i.e., the re-centering step) is equivalent to the update for centers Y using the optimal transport, which in this case is equivalent to the trivial transportation plan that assigns the weight (divided by n) of each unlabeled data in X u to its closest neighbor in centers Y. Algorithm 1 shows the Wasserstein barycenter of the unlabeled data for clustering. while not converged do 6:, t ← t + 1 10: end while 11: a ←â 12: Expectation Step: T ← optimal coupling of p(a, b, M XuY) 14:, balancing coefficients: α, λ, learning rate: β, batch size: b, distance matrix: X, 1: train CNN parameters initially using the labeled data, 2: repeat 3:: Softmax layer outputs on Z l and Z u, 4: {Q 1, ..., Q c} ← cluster on X u using Algorithm. 1, {P 1, ..., P c} ← labeled data grouped to c classes, 6: compute α, β based on amount of the mass in measures Q and P, for each Q i and P j do 8: end for 10: T ← optimal coupling of p(α, β, X), {y u} n u=1 ← pseudo-label data in each cluster Q i with the highest amount of mass transport toward the labeled measure (i.e., argmax T (i, :)), 12: The regular OT problem defined in can be solved by an effective linear programming method in the order of O(n 3 log(n)) time complexity, where n is number of the points in each probability measures. has introduced an interesting approach which relaxes the OT problem by adding a strong convex regularizer to the OT cost function to reduce the time complexity to O(n 2). Specifically, this approach asks for a solution T with more entropy, instead of computing the exact Wasserstein distance. In other words, the regularized OT distances can interpolate the solution, depending on the regularization strength γ, between exact OT (γ = 0), and Maximum Mean Discrepancy, MMD, (γ = ∞). In this work, we use the regularized OT not only for the matter of time complexity, but also it has been shown that the sample complexity of exact Wasserstein distance is O(1/n 1/d), while the regularized Wasserstein distance depending on γ value, is between O(1/ √ n) and O(1/n 1/d), where d is dimension of the samples;. This means that the entropic regularization reduces the chance of over-fitting for our SSL model when it computes the Wasserstein distance between output of the CNN obtained from the labeled and unlabeled data. Hence, our OT problem in the regularized form is recast as follows: where γ is a hyperparameter that balances two terms in, and E(T) = − mn ij T ij (log(T ij − 1) is the entropy of the solution matrix T. It has been shown that if T γ is the solution of the optimization, then ∃!u ∈ R n +, v ∈ R m + such that the solution matrix for is T γ = diag(u)Kdiag(v) where, K = exp(−X/γ). The vectors u and v are updated iteratively between step 1 and 2 by using the well-known Sinkhorn algorithm as follows: step 1)u = a/Kv and step 2)v = b/K u, where/ denotes element-wise division operator. It can be simply shown that there always exists an optimal coupling, π ∈ Π(M, M), that achieves infimum of Eq. in the paper. This is because the cost function ||x − y|| in Eq. is continuous, and based on Theorem 4.1, the existence of an optimal coupling π ∈ Π(R, S) which obtains the infimum is guaranteed due to the tightness of Π(R, S). Furthermore, based on Corollary 6.11, the term W k (x, x) used in Eq. is a continuous function and Π(M, M) is tight again, so the existence of an optimal coupling in Π(M, M) is also guaranteed. Let L 1 be the Lebesgue space of exponent 1, and (X, µ) and (Y, ν) be two Polish probability spaces; let a: X → R ∪ {−∞} and b: Y → R ∪ {−∞} be two upper semi-continuous functions such that Then there is a coupling of (µ, ν) which minimizes the total cost Ec(X, Y) among all possible couplings (X, Y). Lemma 1: Let X and Y be two Polish spaces. Let R ⊂ P(X) and S ⊂ P(Y) be tight subsets of P(X) and P(Y) respectively. Then, the set Π(R, S) of all transference plans whose marginals lie in R and S respectively, is itself tight in P(X × Y). Proof of Lemma: Let µ ∈ R, ν ∈ S, and π ∈ Π(µ, ν). By assuming that, for any > 0 there is a compact set K ⊂ X, independent of the choice of µ in R, such that µ[X nK] ≤; and similarly there is a compact set L ⊂ Y, independent of the choice of ν in S, such that ν[YnL] ≤. Then, for any coupling (X, Y) of (µ, ν), The desired follows because this bound is independent of the coupling, and K × L is compact in X × Y. Lemma 2: Let X and Y be two Polish spaces, and c: X × Y → R ∪ {+∞} a lower semi-continuous cost function. Let h: X × Y → R ∪ {−∞} be an upper semi-continuous function such that c ≥ h. Let (π k) k ∈ N be a sequence of probability measures on X × Y, converging weakly to some Therefore, In particular, if c is non-negative, then F: π → cdπ is lower semi-continuous on P(X × Y), equipped with the topology of weak convergence. Replacing c by c−h, we may assume that c is a non-negative lower semi-continuous function. Then c can be written as the point-wise limit of a non-decreasing family (c) ∈ N of continuous real-valued functions. By monotone convergence,: If X is a Polish space, then a set R ⊂ P(X) is precompact for the weak topology if and only if it is tight, i.e. for any > 0 there is a compact set K such that µ[X nK] ≤ for all µ ∈ R. Proof of Theorem 4.1: Since X is Polish, {µ} is tight in P(X); similarly, {ν} is tight in P(Y). By using the Lemma 1, Π(µ, ν) is tight in P(X × Y), and by using Prokhorovs theorem, this set has a compact closure. By passing to the limit in the equation for marginals, we see that Π(µ, ν) is closed, so it is in fact compact. Then let (π k) k ∈ N be a sequence of probability measures on X × Y, such that cdπ k converges to the infimum transport cost. Extracting a sub-sequence if necessary, we may assume that π k converges to some π ∈ Π(µ, ν)., and c ≥ h by assumption; moreover, hdπ k = hdπ = adµ + bdν; so Lemma 2 implies: Therefore, π is minimizing. Note that further details of the proof of Theorem 4.1 are also available in Villani's book. Corollary 6.11 in Villanis book: ) is a Polish space, and p ∈ [1, ∞), then W p is continuous on P p (X). More explicitly, if µ k (resp. ν k) converges to µ (resp. ν) weakly in P p (X) as k → ∞, then One of the interesting arguments presented in for a standard evaluation of different SSL models is that it may not be feasible to perform model selection for an SSL challenge if the hyperparameters of the model are tuned on the realistically small validation sets. On the other hand, most of the SSL datasets in the literature are designed in such a way that the validation set, which is used for tuning the hyperparameters but not for parameters of the model, is much larger than the training set. For example, the standard SVHN dataset used in our work has about 7000 labeled data in the validation set. Hence, the validation set is seven times larger than the training set of the SSL methods which evaluate their performance by using only 1,000 labeled data during the training. However, this is not a practical choice for a real-world application. This is because, this large validation set will be used as the training set instead of validation set for tuning the hyperparameters. Using small validation sets, however, causes an issue in that the evaluation metric, such as the accuracy for tuning the hyperparameters will be unstable and noisy across the different runs. Although the fact that small validation sets limit the ability for model selection has been discussed in , the work presented in has used the Hoeffding inequality to directly analyze the relationship between the size of validation set and the variance in estimation of a models accuracy: P(|V − E(V)| < p) > 1 − 2 exp(−2np 2). In this inequality, V denotes the empirical estimate of the validation error, E[V] is its hypothetical true value, p is the desired maximum deviation between the estimation and the true value, and n represents the number of samples in the validation set. Based on this inequality, the number of samples in the validation set should be very large. For example, we will require about 20,000 samples in the validation set if we want to be 95% confident in estimation of validation error that differes less than 1% from the absolute true value. Note that in this analysis, validation error is computed as the average of independent binary indicator variables representing if a given sample in the validation set is classified correctly or not. This analysis may be unrealistic because of the assumption that the validation accuracy is the average of independent variables. To address this problem, measure this phenomenon empirically, and train the SSL methods using 1,000 labels in the training set from SVHN dataset and then evaluate them on the validation sets with different sizes. Note that these small synthetic validation sets are generated by different randomly sampled sets without overlapping from the full SVHN validation set. Following the same setting for evaluation of our SSL algorithm (ROT) in a real world scenario, in Fig. 3(a) and Fig. 4(a), we reported the mean and standard deviation of validation errors over five times randomly non-overlapping splitting the SVHN and CIFAR validation sets with varying sizes. The in Fig. 3(a) and Fig. 4(a) indicate that as we increase the size of validation set, the ROT algorithm will be more confident and stable to select its hyperparameters than the case where we use small-size validation set. For a fair comparison between our method and the other SSL methods in Table. 1 of the paper, we have been consistent with other methods in the size of the training and validation sets as it is designed in standard SVHN and CIFAR-10 datasets. Specifically, for SVHN, we used 65,932 images for the training set and 7,325 for the validation set, and for CIFAR-10 dataset, we used 45,000 images for the training set and 5,000 images for the validation set. Fig. 3(b) and Fig. 4(b) indicate the error rate of the ROT algorithm on the SVHN and CIFAR validation sets for different values of λ in our transportation plan. Note that during the tuning of λ, we fixed α in Eq.

Represent the short summary for finding the corresponding scientific article (long)

We propose a new algorithm based on the optimal transport to train a CNN in an SSL fashion.

Represent the TLDR for finding the corresponding scientific article

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.