Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Hashing-based collaborative filtering learns binary vector representations (hash codes) of users and items, such that recommendations can be computed very efficiently using the Hamming distance, which is simply the sum of differing bits between two hash codes. A problem with hashing-based collaborative filtering using the Hamming distance, is that each bit is equally weighted in the distance computation, but in practice some bits might encode more important properties than other bits, where the importance depends on the user. To this end, we propose an end-to-end trainable variational hashing-based collaborative filtering approach that uses the novel concept of self-masking: the user hash code acts as a mask on the items (using the Boolean AND operation), such that it learns to encode which bits are important to the user, rather than the user's preference towards the underlying item property that the bits represent. This allows a binary user-level importance weighting of each item without the need to store additional weights for each user. We experimentally evaluate our approach against state-of-the-art baselines on 4 datasets, and obtain significant gains of up to 12% in NDCG. We also make available an efficient implementation of self-masking, which experimentally yields <4% runtime overhead compared to the standard Hamming distance. Collaborative filtering is an integral part of personalized recommender systems and works by modelling user preference on past item interactions to predict new items the user may like . Early work is based on matrix factorization approaches that learn a mapping to a shared m-dimensional real-valued space between users and items, such that user-item similarity can be estimated by the inner product. The purpose of hashing-based collaborative filtering is the same as traditional collaborative filtering, but allows for fast similarity searches to massively increase efficiency (e.g., realtime brute-force search in a billion items ). This is done by learning semantic hash functions that map users and items into binary vector representations (hash codes) and then using the Hamming distance (the sum of differing bits between two hash codes) to compute user-item similarity. This leads to both large storage reduction (floating point versus binary representations) and massively faster computation through the use of the Hamming distance. One problem with hashing-based collaborative filtering is that each bit is weighted equally when computing the Hamming distance. This is a problem because the importance of each bit in an item hash code might differ between users. The only step towards addressing this problem has been to associate a weight with k-bit blocks of each hash code . However, smaller values of k lead to increased storage cost, but also significantly slower computation due to the need of computing multiple weighted Hamming distances. To solve this problem, without using any additional storage and only a marginal increase in computation time, we present Variational Hashing-based collaborative filtering with Self-Masking (VaHSM-CF). VaHSM-CF is our novel variational deep learning approach for hashing-based collaborative filtering that learns hash codes optimized for selfmasking. Self-masking is a novel technique that we propose in this paper for user-level bit-weighting on all items. Self-masking modifies item hash codes by applying an AND operation between an item and user hash code, before computing the standard Hamming distance between the user and selfmasked item hash codes. Hash codes optimized with self-masking represent which bit-dimensions encode properties that are important for the user (rather than a bitwise -1/1 preference towards each property). In practice, when ranking a set of items for a specific user, self-masking ensures that only bit differences on bit-dimensions that are equal to 1 for the user hash code are considered, while ignoring the ones with a -1 value, thus providing a user-level bitwise binary weigthing. Since selfmasking is applied while having the user and item hash codes in the lowest levels of memory (i.e., register), it only leads to a very marginal efficiency decrease. We contribute (i) a new variational hashing-based collaborative filtering approach, which is optimized for (ii) a novel self-masking technique, that outperforms state-of-the-art baselines by up to 12% in NDCG across 4 different datasets, while experimentally yielding less than 4% runtime overhead compared to the standard Hamming distance. We publicly release the code for our model, as well as an efficient implementation of the Hamming distance with self-masking 1. We focus on collaborative filtering with explicit feedback, which assumes that users and items are related via a user-specified rating: the task is to rank a pool of pre-selected items. This is different from implicit feedback, where the task is to estimate the pool of items that are of interest to the user. Matrix factorization is one of the most successful collaborative filtering methods , but to reduce storage requirements and speed up computation, hashing-based collaborative filtering has been researched. For hashing-based methods the users and items are represented as binary hash codes (as opposed to real-valued vectors), such that the highly efficient Hamming distance (as opposed to the inner product) can be used for computing user-item similarities. Two-stage approaches. Early hashing-based collaborative filtering methods include two stages: First, real-valued user and item vectors are learned, and then the real-valued vectors are transformed into binary hash codes. employ matrix factorization initially, followed by a binary quantization of rounding the real-valued vectors, while ensuring that the hash code is preference preserving of the observed user-item ratings using their proposed Constant Feature Norm constraint. and both explore binary quantization strategies based on orthogonal rotations of the real-valued vectors, which share similarities with Spectral Clustering . However, the two-stage approaches often suffer from large quantization errors , because the hash codes are not learned directly, but rather based on different quantization procedures. Learned hashing approaches. propose Discrete Collaborative Filtering (DCF), which is a binary matrix factorization approach that directly learns the hash codes using relaxed integer optimization, while enforcing bit balancing and decorrelation constraints. Extensions of DCF have focused on incorporating side-information (e.g., reviews associated with a rating); ) and have been redesigned for implicit feedback signals . More recent work addresses the problem that hashing-based collaborative filtering methods have reduced representational power compared to real-valued vectors, but increasing the hash code dimensionality to match the amount of bits used in the real-valued case hurts model generalization . To address this, propose Compositional Coding for Collaborative Filtering (CCCF), which is broadly similar to learning compositional codes for (word) embedding compression . CCCF is a hybrid approach that combines hash codes and real-valued weights: each hash code is split into k blocks of r bits each, and each block is associated with a real-valued scalar indicating the weight of the block. The distance between two CCCF hash codes is then computed as a weighted sum of the Hamming distances of the individual blocks, where each weight is the product of each block's weight. The problem with this approach is that each block requires an individual Hamming distance computation, as well as floating point multiplications of the block weights. In fact, the CCCF block construction no longer allows for highly efficient Boolean operations because the distance computation is weighted by each block's weight. Another problem with CCCF is that it drastically increases storage requirements by needing to store the real-valued weights for all blocks in a hash code. In contrast to CCCF, our proposed variational hashing-based collaborative filtering with selfmasking solves the same problems, as it effectively allows to disable unimportant bits -corresponding to a 1-bit block size with 0/1 weights -without needing to store any additional weights or vectors. Additionally, after having applied the self-masking, user-item similarity can still be computed using only a single Hamming distance on the two hash codes. Hashing-based Collaborative filtering aims to learn binary user and item representations (called hash codes), such that the distance between the representations indicates how well user u likes item i. In practice, the Hamming distance is used due to its fast hardware-level implementation. Formally, we learn z u ∈ {−1, 1} m and z i ∈ {−1, 1} m, where m is the number of bits in the hash code, which is typically chosen to fit into a machine word. The preference of user u for item i is specified by the rating R u,i ∈ {1, 2, 3, ..., K}, where K is the maximum rating, such that the Hamming distance between z u and z i is low when R u,i is high. Computing the Hamming distance computation is extremely efficient due to fast hardware-level implementation of the Boolean operations, as where SUM is computed fast on hardware using the popcnt instruction. Given a user and set of items, the integer-valued Hamming distances can be linear-time sorted using e.g. radix sort (because Hamming distances are bounded in [0, m]) in ascending order to create a ranked list based on user preference . The Hamming distance assigns equal weights to all bits, but in reality bit importance might differ among users. For example, if we consider each bit an encoding of a specific property of an item (e.g., a movie being a thriller), then the weight of each property would be dependent on each user's preference. However, since the hash codes are binary, it is not possible to encode such preference weights without using more storage and computation time due to no longer operating on binary values. In fact, no existing method even allows disabling specific bits for certain users (corresponding to the case of 0 preference weight). We next present a solution for the latter problem, which encodes the importance of each bit directly into the user hash code, and therefore does not require any additional storage. We define the Hamming distance with self-masking: We first apply an AND operation between the user and item hash codes, and then compute the Hamming distance between that and the user hash code. This fundamentally changes the purpose of the user hash code: instead of encoding a positive or negative preference for a property, it encodes which properties are important to the user (-1's from the user hash code are copied to the item due to the AND operation). This allows the model to disable unimportant bits on a user-level, meaning that we enable the model to produce user-specific item representations while still only storing a single hash code for each user and item respectively. Self-masking requires an additional Boolean operation, but since this is applied once the hash codes are already placed in the lowest levels of memory (i.e., register), it only leads to a marginal decrease in efficiency (see Section 4.6 for an empirical analysis). To derive a variational setup for hashing-based collaborative filtering, we define the likelihood of a user, u, and the likelihood of an item i, as the product over the likelihoods of the observed user specified ratings: where θ are the parameters of the (neural network) model. This formulation enforces a dual symmetric effect of users being defined by all their rated items, and items being defined by the ratings provided by all the users. To maximize the likelihood of all observed items and users, we need to maximize the likelihood of the observed ratings p θ (R u,i). Note that instead of maximizing the raw likelihood, we consider the log likelihood to derive the objective below. We assume that the likelihood of a rating, p θ (R u,i), is conditioned on two latent vectors, a user hash code z u, and an item hash code z i. To obtain the hash codes of the user and item, we assume that z u and z i each are sampled by repeating m Bernoulli trials, which have equal probability of sampling -1 and 1. This gives us the following log likelihood, which we wish to maximize: The latent vectors z u and z i are a user and item hash code, and should therefore be conditioned on the user and item respectively. To do this we first multiply and divide with the approximate posterior distributions q φ (z i |i), q ψ (z u |u): where ψ and φ are the parameters of the approximate posteriors. We can now rewrite to an expectation and apply Jensen's inequality to obtain a lower bound on the log likelihood: Since z i and z u will be sampled independently, then q φ (z i |i) and q ψ (z u |u) are independent and we can rewrite to the variational lower bound: where KL(·, ·) is the Kullback-Leibler divergence. Thus, to maximize the expected log likelihood of the observed rating, we need to maximize the conditional log likelihood of the rating, while minimising the KL divergence between the approximate posterior and prior distribution of the two latent vectors. Maximizing the expected conditional log likelihood can be considered as a reconstruction term of the model, while the KL divergence can be considered as a regularizer. Next we present the computation of the approximate posterior distributions q i φ (z i |i) and q u ψ (z u |u) (Section 3.3) and the conditional log likelihood of the rating p θ (R u,i |z u, z i) (Section 3.4). The approximate posterior distributions can be seen as two encoder functions modelled through a neural network by considering the functions as embedding layers. Each encoder function maps either a user or an item to a hash code. Next, we focus on the derivation of the encoder function for the user, as they are both computed in the same way. The probability of the j'th bit is given by: where u is the j'th entry in a learned real-valued embedding E for user u, and σ is the sigmoid function. The j'th bit is then given by: where µ (j) is either chosen stochastically by sampling µ (j) from a uniform distribution in the interval, or chosen deterministically to be 0.5, which can be used for evaluation to obtain fixed hash codes. Note that µ is sampled for each bit. As the sampling is non-differentiable, a straight-through estimator is used for backpropagation. The conditional log likelihood can be considered a reconstruction of the rating, given the user and item hash codes. We model the observed ratings as a ground truth rating with additive standard normally distributed noise, which is then discretized to the observed categorical rating. The conditional log likelihood can then be computed as: where f (z u, z i) is a function that reconstructs the rating given the user and item hash codes. Maximising the log likelihood, log p θ (R u,i |z u, z i), corresponds to minimising the mean squared error (MSE) between R u,i and f (z u, z i), which is done for training the model. Existing work on hashingbased collaborative filtering also employs a MSE objective, and thus implicitly make the same normal distribution assumption as done in this work. We define the reconstruction function to be the self-masking Hamming distance from Eq. 2: where g is a fixed affine transformation that maps the interval of the Hamming distance to the interval of the ratings, such that the minimum and maximum of the Hamming distance correspond to the minimum and maximum of the ratings. The model is now fully differentiable and can be trained end-to-end using backpropagation, such that the network is able to optimize the hash codes directly for self-masking. A depiction of the model is provided in Figure 1. It should be noted that while variational autoencoders are generative models, we do not explicitly utilize this in our model, and are primarily concerned with the reconstruction of the observed ratings. We evaluate on 4 publicly available datasets commonly used in prior work (; ; ; ; and summarized in Table 1 . Specifically, we use: two movie rating datasets, Movielens 1M 2 (ML-1M) and Movielens 10M 3 (ML-10M); a Yelp dataset with ratings of e.g., restaurant and shopping malls 4; and a book rating dataset from Amazon 5 (50% of the ratings are used for testing, 42.5% are used for training, while the last 7.5% are used for validation. We evaluate our method, VaHSM-CF, and all baselines (see Section 4.2) using Normalised Discounted Cumulative Gain (NDCG) (Järvelin & Kekäläinen, 2000), which is often used to evaluate recommender systems with non-binary ratings (or relevance values). We use the average NDCG at cutoffs {2, 6, 10} over all users and report the average for each cutoff value. We use as baselines the state-of-the-art methods for hashing-based collaborative filtering (see Section 2), standard matrix factorisation, and two different strategies for binarising the output of the matrix factorisation 6. We include the standard matrix factorization as a reference to a traditional real-valued collaborative filtering baseline, in order to highlight the performance gap between the hashing-based approaches and a real-valued approach. 7 learns user and item hash codes through a binary matrix factorization solved as a relaxed integer problem. 8 learns hash codes consisting of k blocks, where each block has r bits. A floating point weight is associated with each block for computing user-item similarities as a weighted sum of block-level Hamming distances. In the original paper, the floating point weights are not counted towards the amount of bits used, thus leading to an unfair advantage. For a fair comparison, we count each floating point weight as 16 bits in the following experimental comparison. 9 is the classical matrix factorization based collaborative filtering method, where latent real-valued vectors are learned for users and items. We set the latent dimension to be the same as the number of bits used in the hashing-based approaches. MF mean and MF median are based on MF, but use either each dimension's mean or median for doing the binary quantization to bits . We include these to highlight the large quantization loss occurring when the hash codes are not learned directly. VaH-CF is our proposed method without self-masking. We use it to show the effect of a neural variational hashing-based collaborative approach without self-masking. We train both VaHSM-CF and VaH-CF using the Adam optimizer , and tune the learning rate from the set {0.005, 0.001, 0.0005}, where 0.001 is always chosen across all data sets. The batch size is chosen from the set {100, 200, 400, 800}, where 400 was always chosen. To reduce over-fitting, Gaussian noise is added to the ratings during training, as noise injection has been found beneficial in multiple domains for variational neural models . For the noise injection, we initially set the variance of the Gaussian to 1 and reduce by a factor of 1 − 10 −4 every iteration during training. Our model is implemented using the Tensorflow Python library , and all experiments are run on Titan X GPUs. All hyper parameters for the baselines are tuned using the same set of possible values as in the original papers. For the CCCF baseline, we consider block sizes of {8, 16, 32, 64} and each floating 32 bits ML-1M ML-10M Yelp Amazon NDCG@2 NDCG@6 NDCG@10 NDCG@2 NDCG@6 NDCG@10 NDCG@2 NDCG@6 NDCG@10 NDCG@2 NDCG@6 Table 2: NDCG@K of our method (VaHSM-CF) against baselines using hash codes of length 32 and 64 bits. The missing for CCCF on Amazon are due to the model requiring more than 128 GB of RAM. Statistically significant improvements using a paired two tailed t-test at the 0.05 level, compared to the best existing hashing-based baseline (DCF), are indicated by *. point weight counts for 16 bits. We try all possible combinations that fit within the bit budget, and if a single block is chosen, then the weight is not included in the bit calculation. The are shown in Table 2, for hash code lengths of 32 and 64 bits, as these correspond to common machine word sizes. The highest NDCG per column is shown in bold (among hashingbased baselines), and statistically significantly better than the best hashing-based baseline (DCF), using a paired two tailed t-test at the 0.05 level, are indicated by an asterisk *. The Amazon for CCCF are not included, as the released implementation requires excessive amounts of RAM (>128GB) on this dataset due to the large amount of items and users. Our proposed VaHSM-CF significantly outperforms all hashing-based baselines across all datasets by up to 12%. Our VaH-CF without self-masking is second best (up to 2% better than the hashingbased baselines), although on ML-1M the VaH-CF are not statistically significantly better than DCF. This highlights the benefit of modelling hashing-based collaborative filtering with a variational deep learning based framework, which is notably different from existing hashing-based methods based on dicrete matrix factorization solved as relaxed integer problems. Most importantly however, it shows the significant improvement self-masking brings to hashing-based collaborative filtering. We observe that the top 3 baselines (including our VaH-CF) generally obtain similar scores, which highlights the difficulty of improving performance without changing how the hash codes are used (as done by self-masking). The real-valued MF baseline outperforms all the hashing-based approaches, which is to be expected as the representational power of of 32/64 floating point numbers are notably higher than 32/64 bits. However, our VaHSM-CF bridges a large part of the gap between existing hashing-based methods and MF, such that the NDCG difference in most cases is below 0.01. Additionally, the large performance decrease by both the mean or median rounding shows the large quantization error obtained if the hash codes are not learned directly, as done by our and the existing hashing-based approaches. How self-masking influences the convergence rate of the model. Figure 2a shows the convergence rate for the ML-1M dataset with and without self-masking. We see that training with self-masking significantly improves the convergence rate compared to the model without self-masking. Since the time for a single epoch is approximately the same with and without self-masking, we conclude that self-masking not only improves NDCG, but also reduces training time by a very large margin. Figure 2: 2a shows convergence using the validation NDCG@10 on ML-1M, where self-masking significantly speeds up convergence. We observe the same trend on the other datasets (see Appendix A.1). 2b Shows the test NDCG@10 when varying whether hash codes are sampled stochastically or deterministically while training and for evaluation. For example, Det. Eval + Stoc. Train corresponds to deterministic sampling of hash codes for evaluation, while sampling stochastically when training the model. Convergence plots for the remaining datasets are shown in the Appendix, where we observe the same trend. Stochastic or deterministic sampling. We investigate the effect of the sampling strategy for the hash codes (see Eq. 10) during training and evaluation. The sampling can either be deterministic (µ (j) = 0.5) or stochastic (µ (j) is sampled uniformly at random from), and does not have to be the same for training and evaluation. Figure 2b shows the performance for these 4 configurations across all datasets. We see that stochastic training with deterministic evaluation performs the best, while deterministic training and deterministic evaluation perform second best. As expected, stochastic sampling at evaluation performs significantly worse than deterministic sampling, as every item has a very small probability of being sampled such that it has a small Hamming distance to a user, even though it has a low rating (and vice versa for highly rated items). 4.6 RUNTIME ANALYSIS Self-masking has an additional cost added to the standard Hamming distance, due to the additional AND operation between the user and item hash codes (see Eq. 1 and 2). We now investigate the actual runtime cost associated with this modification. We implement both the Hamming distance and Hamming distance with self-masking efficiently in C on a machine with a 64 bit instruction set. A test environment was made with 64 bit hash codes for 100,000 users and 100,000 items. For each user, the distances were computed to all items using both the Hamming distance and Hamming distance with self-masking. We measure the actual time taken for computing the distances to all items from each user, and report the average over 50 repeated runs. All experiments are run on a single thread 10, with all users and items loaded in RAM. The code was compiled with the highest optimization level, and utilizing all optimization flags applicable to the hardware. We verified the produced assembler code used the efficient popcnt instruction. The mean experiment time was 8.0358s when using the Hamming distance, and 8.3506s when using the Hamming distance with self-masking. Thus, self-masking only adds a runtime overhead of 3.91% compared to using the standard Hamming distance. As in this setup we are only computing the distances, this can be seen as an upper bound of the actual overhead in a complete system, as the remaining operations (e.g., sorting) would be the same with and without self-masking. Thus, this provides a good trade-off compared to the large performance gains it yields. Note that the measured times are for the total 10 10 distance computations, highlighting the scalability of hashing-based methods to datasets of massive scale. For comparison, if the experiment is repeated with computing the dot product of floating point vectors of size 64, then the computation time is 648.5632s, thus close to 80x slower. We proposed an end-to-end trainable variational hashing-based collaborative filtering method, which optimizes hash codes using a novel modification to the Hamming distance, which we call selfmasking. The Hamming distance with self-masking first creates a modified item hash code, by applying an AND operation between the user and item hash codes, before computing the Hamming distance. Intuitively, this can be seen as ignoring user-specified bits when computing the Hamming distance, corresponding to applying a binary importance weight to each bit, but without using more storage and only a very marginal runtime overhead. We verified experimentally that our model outperforms state-of-the-art baselines by up to 12% in NDCG at different cutoffs, across 4 widely used datasets. These gains come at a minimal cost in recommendation time (self-masking only increased computation time by less than 4%). A.1 CONVERGENCE PLOTS Convergence plots for Yelp, Amazon, and ML-10M are shown in Figure 3. We observe a similar trend to ML-1M in Figure 2a, where the self-masking leads to a notably faster rate of convergence.

Represent the short summary for finding the corresponding scientific article (long)

We propose a new variational hashing-based collaborative filtering approach optimized for a novel self-mask variant of the Hamming distance, which outperforms state-of-the-art by up to 12% on NDCG.

Represent the TLDR for finding the corresponding scientific article


Question: Represent the scientific article to find a short TLDR.

Question: Determining the appropriate batch size for mini-batch gradient descent is always time consuming as it often relies on grid search. This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit that achieves performance equivalent to that of best fixed batch-size. At each epoch, the RMGD samples a batch size according to a certain probability distribution proportional to a batch being successful in reducing the loss function. Sampling from this probability provides a mechanism for exploring different batch size and exploiting batch sizes with history of success. After obtaining the validation loss at each epoch with the sampled batch size, the probability distribution is updated to incorporate the effectiveness of the sampled batch size. Experimental show that the RMGD achieves performance better than the best performing single batch size. It is surprising that the RMGD achieves better performance than grid search. Furthermore, it attains this performance in a shorter amount of time than grid search. Gradient descent (GD) is a common optimization algorithm for finding the minimum of the expected loss. It takes iterative steps proportional to the negative gradient of the loss function at each iteration. It is based on the observation that if the multi-variable loss functions f (w) is differentiable at point w, then f (w) decreases fastest in the direction of the negative gradient of f at w, i.e., −∇f (w). The model parameters are updated iteratively in GD as follows: DISPLAYFORM0 where w t, g t, and η t are the model parameters, gradients of f with respect to w, and learning rate at time t respectively. For small enough η t, f (w t) ≥ f (w t+1) and ultimately the sequence of w t will move down toward a local minimum. For a convex loss function, GD is guaranteed to converge to a global minimum with an appropriate learning rate. There are various issues to consider in gradient-based optimization. First, GD can be extremely slow and impractical for large dataset: gradients of all the data have to be evaluated for each iteration. With larger data size, the convergence rate, the computational cost and memory become critical, and special care is required to minimize these factors. Second, for non-convex function which is often encountered in deep learning, GD can get stuck in a local minimum without the hope of escaping. Third, stochastic gradient descent (SGD), which is based on the gradient of a single training sample, has large gradient variance, and it requires a large number of iterations. This ultimately translates to slow convergence. Mini-batch gradient descent (MGD), which is based on the gradient over a small batch of training data, trades off between the robustness of SGD and the stability of GD. There are three advantages for using MGD over GD and SGD: 1) The batching allows both the efficiency of memory usage and implementations; 2) The model update frequency is higher than GD which allows for a more robust convergence avoiding local minimum; 3) MGD requires less iteration per epoch and provides a more stable update than SGD. For these reasons, MGD has been a popular algorithm for machine learning. However, selecting an appropriate batch size is difficult. Various studies suggest that there is a close link between performance and batch size used in;;.There are various guidelines for selecting a batch size but have not been completely practical BID1. Grid search is a popular method but it comes at the expense of search time. There are a small number of adaptive MGD algorithms to replace grid search BID3; BID4. These algorithms increase the batch size gradually according to their own criterion. However, these algorithms are based on convex loss function and hard to be applied to deep learning. For non-convex optimization, it is difficult to determine the optimal batch size for best performance. This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit for achieving best performance in grid search by selecting an appropriate batch size at each epoch with a probability defined as a function of its previous success/failure. At each epoch, RMGD samples a batch size from its probability distribution, then uses the selected batch size for mini-batch gradient descent. After obtaining the validation loss at each epoch, the probability distribution is updated to incorporate the effectiveness of the sampled batch size. The benefit of RMGD is that it avoids the need for cumbersome grid search to achieve best performance and that it is simple enough to apply to any optimization algorithm using MGD. The detailed algorithm of RMGD are described in Section 4, and experimental are presented in Section 5. There are only a few published on the topic of batch size. It was empirically shown that SGD converged faster than GD on a large speech recognition database. It was determined that the range of learning rate ing in low test errors was considerably getting smaller as the batch size increased on convolutional neural networks and that small batch size yielded the best test error, while large batch size could not yield comparable low error rate BID2. It was observed that larger batch size are more liable to converge to a sharp local minimum thus leading to poor generalization. It was found that the learning rate and the batch size controlled the trade-off between the depth and width of the minima in MGD Jastrzkebski et al.. A small number of adaptive MGD algorithms have been proposed. BID3 introduced a methodology for using varying sample size in MGD. A relatively small batch size is chosen at the start, then the algorithm chooses a larger batch size when the optimization step does not produce improvement in the target objective function. They assumed that using a small batch size allowed rapid progress in the early stages, while a larger batch size yielded high accuracy. However, this assumption did not corresponded with later researches that reported the degradation of performance with large batch size;;. Another similar adaptive algorithm, which increases the batch size gradually as the iteration proceeded, was done by. The algorithm uses relatively few samples to approximate the gradient, and gradually increase the number of samples with a constant learning rate. It was observed that increasing the batch size is more effective than decaying the learning rate for reducing the number of iterations. However, these increasing batch size algorithms lack flexibility since it is unidirectional. BID0 proposed a dynamic batch size adaptation algorithm. It estimates the variance of the stochastic gradients and adapts the batch size to decrease the variance. However, this algorithm needs to find the gradient variance and its computation depends on the number of model parameters. Batch size can also be considered as a hyperparameter, and there have been some proposals based on bandit-based hyperparameter (but not batch size) optimization which maybe applicable for determining the best fixed batch size. introduced a successive halving algorithm. This algorithm uniformly allocates a budget to a set of hyperparameter configurations, evaluates the performance of all configurations, and throws out the worst half until one configuration remains. introduced a novel bandit-based hyperparameter optimization algorithm referred as HYPERBAND. This algorithm considers the optimization problem as a resource allocation problem. The two algorithms mentioned above are not adaptive, and for searching a small hyperparameter space, the two algorithms will not be very effective. The experimental in this paper show that adaptive MGD tends to perform better than fixed MGD.Figure 1: An overall framework of considered resizable mini-batch gradient descent algorithm (RMGD). The RMGD samples a batch size from a probability distribution, and parameters are updated by mini-batch gradient using the selected batch size. Then the probability distribution is updated by checking the validation loss.3 SETUP DISPLAYFORM0 be the set of possible batch size and π = {π k} K k=1 be the probability distribution of batch size where b k, π k, and K are the k th batch size, the probability of b k to be selected, and number of batch sizes respectively. This paper considers algorithm for multi-armed bandit over B according to Algorithm 1. Let w τ ∈ W be the model parameters at epoch τ, andw t be the temporal parameters at sub iteration t. Let J: W → R be the training loss function and let g = ∇J(w) be the gradients of training loss function with respect to the model parameters. η τ is the learning rate at epoch τ. Let: W → R be the validation loss function, and y k ∈ {0, 1} be the cost of choosing the batch size b k. In here, y k = 0 if the validation loss decreases by the selected batch size b k (well-updating) and y k = 1 otherwise (misupdating). The aim of the algorithm is to have low misupdating. For the cost function y k, graduated losses such as hinge loss and percentage of nonnegative changes in validation loss can be variations of 0-1 loss. However, there are no differences in regret bound among them in this setting and it is experimentally confirmed that there are little performance gaps among them. Therefore, this paper introduces the 0-1 loss, which is simple and basic. The resizable mini-batch gradient descent (RMGD) sets the batch sizes as multi arms, and at each epoch it samples one of the batch sizes from probability distribution. Then, it suffers a cost of selecting this batch size. Using the cost, probability distribution is updated. The overall framework of the RMGD algorithm is shown in Figure 1. The RMGD consists of two components: batch size selector and parameter optimizer. The selector samples a batch size from probability distribution and updates the distribution. The optimizer is usual mini-batch gradient. Selector samples a batch size b kτ ∈ B from the probability distribution π τ at each epoch τ where k τ is selected index. Here b k is associated with probability π k. The selected batch size b kτ is applied to optimizer for MGD at each epoch, and the selector gets cost y kτ from optimizer. Then, the selector Input: DISPLAYFORM0: Set of batch sizes π 0 = {1/K, . . ., 1/K}: Prior probability distribution 1: Initialize model parameters w 0 2: for epoch τ = 0, 1, 2,... Select batch size b kτ ∈ B from π τ Set temporal parametersw 0 = w τ 5: DISPLAYFORM0 Compute gradient g t = ∇J(w t)7: DISPLAYFORM1 end for DISPLAYFORM2 Update w τ +1 =w T Observe validation loss (w τ +1): DISPLAYFORM0 Get cost y kτ = 0 13: DISPLAYFORM1 Get cost y kτ = 1 15: 16: DISPLAYFORM0 Set temporal probabilityπ i = π DISPLAYFORM1 updates probabilities by randomized weighted majority, DISPLAYFORM2 Optimizer updates the model parameters w. For each epoch, temporal parametersw 0 is set to w τ, and MGD iterates T = m/b kτ 1 times using the selected batch size b kτ where m is the total number of training samples:w DISPLAYFORM3 After T iterations at epoch τ, the model parameters is updated as w τ +1 =w T. Then, the optimizer obtains validation loss, and outputs cost as follows: DISPLAYFORM4 The RMGD samples an appropriate batch size from a probability distribution at each epoch. This probability distribution encourages exploration of different batch size and then later exploits batch size with history of success, which means decreasing validation loss. FIG1 shows an example of training progress of RMGD. The figure represents the probability distribution with respect to epoch. The white dot represents the selected batch size at each epoch. In the early stage of training, 1 x is the least integer that is greater than or equal to x commonly, all batch sizes tend to decrease validation loss: π is uniform. Thus, all batch size have equal probability of being sampled (exploration). In the later stages of training, the probability distribution varies based on success and failure. Thus, better performing batch size gets higher probability to be sampled (exploitation). In this case, 256 is the best performing batch size. The regret bound of the RMGD follows the regret bound derived in. The goal of this algorithm is to have low regret for not selecting the best performing batch size such that DISPLAYFORM0 where the expectation is over the algorithm's randomness of batch size selection and the second term on the right-hand side is the cumulative sum of the cost by the best fixed batch size which minimizes the cumulative sum of the cost. The regret of the RMGD is bounded, DISPLAYFORM1 In particular, setting β = log(K)/(KT), the regret is bounded by 2 K log(K)T, which is sublinear with T. The detailed derivation of regret bound is described in the appendix A. This section describes various experimental on MNIST, CIFAR10, and CIFAR100 dataset. In the experiments, simple convolutional neural networks (CNN) is used for MNIST and'All-CNN-C' Springenberg et al. FORMULA14 is used for CIFAR10 and CIFAR100. The details of the dataset and experimental settings are presented in the appendix B. The validity of the RMGD was assessed by performing image classification on the MNIST dataset using AdamOptimizer and AdagradOptimizer as optimizer. The experiments were repeated 100 times for each algorithm and each optimizer, then the were analyzed for significance. FIG2 shows the probability distribution and the selected batch size with respect to epoch during training for the RMGD. The white dot represents the batch size selected at each epoch. The top figure is the case that small batch size performs better. After epoch 50, batch size 32 gets high probability and is selected more than others. It means that batch size 32 has less misupdating in this case. The gradually increasing batch size algorithm may not perform well in this case. The middle figure is the case that large batch size performs better. After epoch 60, batch size 512 gets high probability and selected more than others. The bottom figure shows that the best performing batch size varies with epoch. During epoch from 40 to 55, batch size of 256 performs best, and best performing batch size switches to 128 during epoch from 60 to 70, then better performing batch size backs to 256 after epoch 80. In the , any batch size can be a successful batch size in the later stages without any particular order. The RMGD is more flexible for such situation than the MGD or directional adaptive MGD such as gradually increasing batch size algorithm. FIG3 shows the test accuracy of each algorithm. The error bar is standard error. The number in parenthesis next to MGD represents the batch size used in the MGD.' Basic','sub','super','hinge', and'ratio' in parenthesis next to RMGD represent RMGD settings'batch size set equal to grid search, 0-1 loss','subset of basic, 0-1 loss','superset of basic, 0-1 loss','basic set, hinge loss', and'basic set, percentage of non-negative changes in validation loss', respectively. The left figure is the test accuracy with AdamOptimizer. The right figure is the test accuracy with AdagradOptimizer. Among the MGD algorithms, relatively small batch sizes lead to higher performance than large batch sizes and batch size 64 achieves the best performance in grid search. These Most RMGD settings outperform all fixed MGD algorithms in both case. Although the performance of RMGD is not significantly increased compared to the best MGD, the purpose of this algorithm is not to improve performance, but to ensure that the best performance is achieved without performing a grid search on the batch size. Rather, the improved performance of the RMGD is a surprising . Therefore, the RMGD is said to be valid. There are little performance gap among RMGD settings. The'sub' setting outperforms the'basic' setting in left figure, but the opposite is shown in right figure. Therefore, there is no clear tendency of performance change depending on the size of the batch size set. TAB0 and 2 present iterations and real time for training, mean, maximum, and minimum of test accuracies for each algorithm with AdamOptimizer and AdagradOptimizer respectively. The MGD (total) is the summation of the iterations and real time of whole MGDs for grid search. The RMGD (basic) outperforms best performing MGD and is, also, faster than best performing MGD. Furthermore, it is 8 times faster than grid search in both cases. In the , the RMGD is effective regardless of the optimizer. The CIFAR10 and CIFAR100 dataset were, also, used to assess effectiveness of the RMGD. The experiments were repeated 25 times and 10 times, respectively. In these experiments, all images are whitened and contrast normalized before being input to the network. FIG4 shows the test accuracy for each algorithm. The left figure represents the test accuracy on CIFAR10. In contrast to the MNIST , relatively large batch sizes lead to higher performance than small batch sizes and batch size 256 achieves the best performance in grid search. The right figure represents the test accuracy on CIFAR100 and batch size 128 achieves the best performance in grid search. The on MNIST, CIFAR10 and CIFAR100 indicate that it is difficult to know which batch size is optimal before performing a grid search. Meanwhile, all RMGD settings have again exceeded the best performance of fixed MGD. There are no significant performance gaps among RMGD settings, so there is no need to worry about choosing appropriate batch size set or selecting cost function. Table 3 and 4 present the detailed on CIFAR10 and CIFAR100 dataset. The RMGD (basic) is a little slower than single best performing MGD (256 for CIFAR10 and 128 for CIFAR100), however, it was much faster than grid search -about 4.6 times on CIFAR10 and 5.0 times on CIFAR100 faster. Therefore, this , also, show the effectiveness of the RMGD.It is difficult to compare the RMGD with other adaptive batch size algorithm, e.g. coupling adaptive batch sizes (CABS) BID0, directly since the underlying goals are different. While the goal of the RMGD is to reduce the validation loss in terms of generalization performance, the CABS determines the batch size to balance between the gradient variance and computation. However, it is obvious that the RMGD is simpler and easier to implement than any other adaptive algorithm cited in this paper, and comparing the test accuracy between the RMGD and the CABS on the CIFAR10 and CIFAR100 using the same experimental settings with'All-CNN-C' shows that the performance of the RMGD is higher than that of the CABS (CIFAR10: 87.862 ± 0.142, CIFAR100: 60.782 ± 0.421). And again, the purpose of this algorithm is not to outperform other algorithms, but to guarantee that the best performance is reached without grid search. Selecting batch size affects the model quality and training efficiency, and determining the appropriate batch size is time consuming and requires considerable resources as it often relies on grid search. The focus of this paper is to design a simple robust algorithm that is theoretically sound and applicable in many situations. This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multiarmed bandit that achieves equivalent performance to that of best fixed batch-size. At each epoch, the RMGD samples a batch size according to certain probability distribution of a batch being successful in reducing the loss function. Sampling from this probability provides a mechanism for exploring different batch size and exploiting batch sizes with history of success. After obtaining the validation loss at each epoch with the sampled batch size, the probability distribution is updated to incorporate the effectiveness of the sampled batch size. The goal of this algorithm is not to achieve state-of-the-art accuracy but rather to select appropriate batch size which leads low misupdating and performs better. The RMGD essentially assists the learning process to explore the possible domain of the batch size and exploit successful batch size. The benefit of RMGD is that it avoids the need for cumbersome grid search to achieve best performance and that it is simple enough to apply to various field of machine learning including deep learning using MGD. Experimental show that the RMGD achieves the best grid search performance on various dataset, networks, and optimizers. Furthermore, it, obviously, attains this performance in a shorter amount of time than the grid search. Also, there is no need to worry about which batch size set or cost function to choose when setting RMGD. In , the RMGD is effective and flexible mini-batch gradient descent algorithm. In the RMGD algorithm, there are K batch sizes as multi arms with the probability distribution π ∈ S, and at each epoch the algorithm should select one of the batch sizes b kτ. Then it receives a cost of selecting this arm, y kτ τ ∈ {0, 1} by testing the validation loss. The vector y τ ∈ {0, 1} K represents the selecting cost for each batch size. The goal of this algorithm is to have low regret for not selecting the best performing batch size. DISPLAYFORM0 where the expectation is over the algorithm's randomness of batch size selection. Let S be the probability simplex, the selecting loss functions be f τ (π) = π, y τ 2 and R: S → R be a regularization function that is often chosen to be strongly convex with respect to some norm || · ||. The algorithm select a batch size with probability P[b kτ] = π kτ τ and therefore f τ (π τ) is the expected cost of the selected batch size at epoch τ. The gradient of the selecting loss function is y τ. However, only one element y kτ τ is known at each epoch. To estimate gradient, random vector z τ is defined as follows: DISPLAYFORM1 and expectation of z τ satisfies, DISPLAYFORM2 The most natural learning rule is to set the probability distribution which has minimal cost on all past epochs. It is referred to as Follow-the-Regularized-Leader (FTRL) in online learning: DISPLAYFORM3 where β is positive hyperparameter. The FTRL has a problem that it requires solving an optimization problem at each epoch. To solve this problem, Online Mirror Descent (OMD) is applied. The OMD computes the current probability distribution iteratively based on a gradient update rule and the previous probability distribution and lies in the update being carried out in a'dual' space, defined by regularizer. This follows from considering ∇R as a mapping from R K onto itself. The OMD relies on Bregman divergence. The Bregman divergence between π andπ with respect to the regularizer R is given as: DISPLAYFORM4 and a Bregman projection ofπ onto simplex S: DISPLAYFORM5 Then the probability distribution is updated by the OMD as follows: DISPLAYFORM6 In general, if R is strongly convex, then ∇R becomes a bijective mapping, thusπ τ +1 can be recovered by the inverse gradient mapping (∇R) −1. Given that R is strongly convex, the OMD and FTRL produce equivalent predictions: DISPLAYFORM7 2 π, y is the inner product between vectors π and y DISPLAYFORM8 Therefore, the regret of the RMGD is bounded, DISPLAYFORM9 In particular, setting β = log(K)/(KT), the regret is bounded by 2 K log(K)T, which is sublinear with T. MNIST is a dataset of handwritten digits that is commonly used for image classification. Each sample is a black and white image and 28 × 28 in size. The MNIST is split into three parts: 55,000 samples for training, 5,000 samples for validation, and 10,000 samples for test. CIFAR10 consists of 60,000 32 × 32 color images in 10 classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck), with 6,000 images per class. The CIFAR10 is split into three parts: 45,000 samples for training, 5,000 samples for validation, and 10,000 samples for test. CIFAR100 consists of 60,000 32 × 32 color images in 100 classes. The CIFAR100 is split into three parts: 45,000 samples for training, 5,000 samples for validation, and 10,000 samples for test. The simple CNN consists of two convolution layers with 5 × 5 filter and 1 × 1 stride, two max pooling layers with 2 × 2 kernel and 2 × 2 stride, single fully-connected layer, and softmax classifier. Description of the'All-CNN-C' is provided in TAB3. For MNIST, AdamOptimizer with η = 10 −4 and AdagradOptimizer with η = 0.1 are used as optimizer. The basic batch size set B = {16, 32, 64, 128, 256, 512}, subset of basic B − = {16, 64, 256}, and superset of basic, 24, 32, 48, 64, 96, 128, 192, 256, 384, 512}. The model is trained for a total of 100 epochs. For CIFAR10 and CIFAR100, MomentumOptimizer with fixed momentum of 0.9 is used as optimizer. The learning rate η k is scaled up proportionately to the batch size (η k = 0.05 * b k /256) and decayed by a schedule S = in which η k is multiplied by a fixed multiplier of 0.1 after 200, 250, and 300 epochs respectively. The model is trained for a total of 350 epochs. Dropout is applied to the input image as well as after each convolution layer with stride 2. The dropout probabilities are 20% for dropping out inputs and 50% otherwise. The model is regularized with weight decay λ = 0.001. The basic batch size set B = {16, 32, 62, 128, 256}, subset of basic B − = {16, 64, 256}, and superset of basic B + = {16, 24, 32, 48, 64, 96, 128, 192, 256}. For all experiments, rectified linear unit (ReLU) is used as activation function. For RMGD, β is set to log/(6 * 100) ≈ 0.055 for MNIST and log/(5 * 350) ≈ 0.030 for CIFAR10 and CIFAR100. The basic batch size selecting cost is 0-1 loss, hinge loss is max{0, τ − τ −1}, and ratio loss is max{0, ( τ − τ −1)/ τ −1 }. 1 × 1 conv. 10 or 100 ReLU, stride 1 pool averaging over 6 × 6 spatial dimensions softmax 10-way or 100-way softmax DISPLAYFORM0

Represent this text to find a much much longer article about it with say hundreds-thousands of words

An optimization algorithm that explores various batch sizes based on probability and automatically exploits successful batch size which minimizes validation loss.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the article for finding a single-sentence TLDR.

Question: Knowledge graph has gained increasing attention in recent years for its successful applications of numerous tasks. Despite the rapid growth of knowledge construction, knowledge graphs still suffer from severe incompletion and inevitably involve various kinds of errors. Several attempts have been made to complete knowledge graph as well as to detect noise. However, none of them considers unifying these two tasks even though they are inter-dependent and can mutually boost the performance of each other. In this paper, we proposed to jointly combine these two tasks with a unified Generative Adversarial Networks (GAN) framework to learn noise-aware knowledge graph embedding. Extensive experiments have demonstrated that our approach is superior to existing state-of-the-art algorithms both in regard to knowledge graph completion and error detection. Knowledge graph, as a well-structured effective representation of knowledge, plays a pivotal role in many real-world applications such as web search , question answering , and personalized recommendation . It is constructed by extracting information as the form of triple from unstructured text using information extraction systems. Each triple (h, r, t) represents a relation r between a head entity h and a tail entity t. Recent years have witnessed extensive construction of knowledge graph, such as Freebase , DBPedia , and YAGO . However, these knowledge graphs suffer from severe sparsity as we can never collect all the information. Moreover, due to the huge volumes of web resources, the task to construct knowledge graph usually involves automatic mechanisms to avoid human supervision and thus inevitably introduces many kinds of errors, including ambiguous, conflicting and erroneous and redundant information. To address these shortcomings, various methods for knowledge graph refinement have been proposed, whose goals can be arguably classified into two categories: knowledge graph completion, the task to add missing knowledge to the knowledge graph, and error detection, the task to identify incorrect triples in the knowledge graph. Knowledge graph embedding (KGE) currently hold the state-of-the-art in knowledge graph completion for their promising . Nonetheless, they highly rely on high quality training data and thus are lack of robustness to noise . Error detection in knowledge graph is a challenging problem due to the difficulty of obtaining noisy data. Reasoning based methods are the most widely used methods for this task . Without the guidance of noisy data, they detect errors by performing reasoning over the knowledge graph to determine the correctness of a triple. A rich ontology information is required for such kind of methods and thus impede its application for real-world knowledge graphs. Existing works consider knowledge graph embedding and error detection independently whereas these two tasks are inter-dependent and can greatly influence each other. On one hand, error detection model is extremely useful to prepare reliable data for knowledge graph embedding. On the other hand, high quality embedding learned by KGE model provides a basis for reasoning to identify noisy data. Inspired by the recent advances of generative adversarial deep models , in this paper, we proposed to jointly combine these two tasks with a unified GAN framework, known as NoiGAN, to learn noise-aware knowledge graph embedding. In general, NoiGAN consists of two main components, a noise-aware KGE model to learn robuster representation of knowledge and an adversarial learning framework for error detection. During the training, noiseaware KGE model takes the confidence score learned by GAN as guidance to eliminate the noisy data from the learning process whereas the GAN requires that KGE model continuously provides high quality embedding as well as credible positive examples to model the discriminator and the generator. Cooperation between the two components drives both to improve their capability. The main contributions of this paper are summarized as follows: • We propose a unified generative adversarial framework NoiGAN, to learn noise-aware knowledge graph embedding. Under the framework, the KGE model and error detection model could benefit from each other: the error detection model prepares reliable data for KGE model to improve the quality of embedding it learns, while the KGE model provides a promising reasoning model for the error detection model to better distinguish noisy triples from the correct one. • Our proposed framework can be easily generalized to various KGE models to enhance their ability in dealing with noisy knowledge graph. • We experimentally demonstrate that our new algorithm is superior to existing state-of-the-art algorithms. The KGE model and GAN can alternately and iteratively boost performance in terms of both knowledge graph completion and noise detection. Embedding based methods currently hold the state-of-the-art in knowledge graph completion for their promising (b). They aim to capture the similarity of entities by embedding entities and relations into continuous low-dimensional vectors. Existing methods can be roughly divided into two categories: translational distance models and semantic matching models. Translational distance models measure the plausibility of a fact as the distance between two entities after a translation carried out by the relation. TransE , TransH and TransR are the representative approaches in this category. Semantic matching models measure plausibility of facts by matching latent semantics of entities and relations embodied in their vector space representations. The typical models include RESCAL , DistMult and ComplEx . To optimize the KGE model,negative sampling is usually required to minimize the margin based ranking loss. A conventional method to construct negative samples is randomly sampling. However, negative samples generated through a random mode are often too easy to be discriminated from positive facts and thus make little contribute towards the training. Some recent works proposed to incorporate GAN for better negative sampling to improve the quality of embeddings . Nonetheless, none of the above methods has taken the potential noisy data into consideration, which leads to their sensitivity to unreliable data . In this paper, we proposed a novel technique to enable current embedding models to cope with noisy data. Due to lack of noisy data samples, error detection in knowledge graph is a challenging task. Existing methods can be either ontology based or anomaly detection based. Ontology based methods address this problem by exploring additional ontology information. A larger number of ontology reasoners are developed to utilize logic programming to derive uncovering contradictions among observed facts (; ;). Rich ontology information is required for such kind of methods and thus impede its application to real-world knowledge graphs. Another kind of methods is anomaly detection based methods . The main drawback of anomaly detection based methods are that they do not necessarily identify errors, but also natural outliers, which will compromise the objectivity of its . More recently, a novel confidence-aware framework proposed to incorporate triple confidence into KGE model to detect noises while learning knowledge representations simultaneously. However, it measures the triple confidence merely based on the how well the triple fits the model, which is easily affected by model bias. In this section, we illustrate our proposed framework, NoiGAN, in details. The goal of our proposed framework is to learn noise-aware KG embedding with Generative Adversarial Networks (GAN). As shown in Figure 1, it consists of three components: (i) Noise aware KGE model, which incorporates a confidence score C(h, r, t) into the KGE model to isolate the impact of noise over embedding vectors. Meanwhile, it provides high quality embedding to model discriminator and generator (Section 3.1); (ii) Triple generator G((h, r, t)|(h, r, t); θ G ), which learns to generate the most likely triples to be the noisy triples by corrupting the correct triple (h, r, t). It aims at providing reliable noisy data for the discriminator as well as noise aware KGE model (Section 3.2.1); (iii) Discriminator D((h, r, t); θ D ), which tries to distinguish correct triples from noisy triples, and assigns a confidence score for each triple fact to describe the correctness of it. The discriminator is used to determine confidence score C(h, r, t) in the noise aware KGE model. It also naturally provides guidance to the generator to produce higher quality noisy data (Section 3.2.2). Before further discussing the detail of the algorithm, we first give a formal definition to knowledge graph. A knowledge graph, denoted by G = {E, R, T}, consists of a set of entities E, a set of relations R and a set of of observed facts T = {(h, r, t) | h, t ∈ E, r ∈ R}. A fact is represented by a triple (h, r, t), where h, t and r denote head entity, tail entity, and relation, respectively. To learn a more robust representation for noisy knowledge graph, we propose a noise-aware KGE model to eliminate the impact of noise over embedding vectors by incorporating confidence score into KGE model. It can be easily adapted to any KGE models. For example, we can follow TransE to represent entities and relations. Given a fact (h, r, t), TransE aims to learn corresponding low dimensional vectors h, r and t in the same space for h, r and t, so that the distance between h + r and t is minimized. The scoring function of TransE is then defined as f r (h, t) = h + r − t 2 1. To optimize the KGE model, we use a loss function similar to the negative sampling loss in according to . where γ is the margin, σ is the sigmoid function, T represents the observed facts from the knowledge graph and (h, r, t) is the negative sample for triple (h, r, t). To reduce the effects of randomness, we sample multiple negative triples for each observed fact. We denote the negative triples set of a triple (h, r, t) as N (h, r, t). Negative triples set is constructed by replacing the head or tail entity of the observed triples with an entity sampled randomly from entity set E: A vast majority of existing embedding models assume that all triple facts hold true to knowledge graph, which is inappropriate. In fact, knowledge graph contains many kinds of errors (e.g., ambiguous, conflicting, erroneous, redundant) due to the automatic construction process. To isolate the impact of noise over embedding vectors, following , we adopted the concept confidence to describe whether a triple fact is noisy or not. In , confidence of a triple is measured by how well the triple fits to the KGE model. A central issue with such measurement is that it is easily affected by model bias and thus lead to unreliability. Unlike , we learned the confidence scores using a discriminator, which can be more impartial and promising. We will discuss how to obtain such discriminator in Section 3.2.2. With the introduction of confidence, we can eliminate the noisy data from the learning process of KGE model. where C(h, r, t) is the confidence score assigned to each positive triple (h, r, t). To be specific, C(h, r, t) can be both a binary variable or a soft value from the interval. We denote the previous one as the hard version and the later one as the soft version. After introducing our noise aware KGE model, we now show how to identify the error in knowledge graph to enhance embedding quality. Typically label information about noisy triples is costly and labor intensive to obtain, which is the central challenge to the task of error detection. Without the guidance of noisy data, previous works propose to leverage ontology information to exploit reasoning for inconsistency detection. However, these ontology information can also be difficult to gain. To address this problem, inspired by recent advance of GAN, we propose a novel end-to-end adversarial learning framework to detect noise in knowledge graph. The proposed GAN cooperates with the noise aware KGE model in an interactive way. GAN utilizes the embedding learned by the noise aware KGE model to distinguish true triples and noisy triples. Meanwhile the noise-aware KGE model requires GAN to learn a confidence score so as to eliminate the impact of noise over embedding vectors. Our adversarial learning framework consists of two components, a generator and a discriminator. Unlike traditional GAN whose ultimate goal is to train a good generator, we aim to produce a good discriminator to distinguish noisy triples from true triples. To address noisy data deficiency issue, the generator and the discriminator act as two players in a minimax game: the generator learns to continually generate "more confusing" triples and thus provides better quality noisy data for the discriminator, whereas the discriminator is trained to draw a clear distinction between the true triples and the noisy triples generated by its opponent generator. Formally, it can be formulated as follows: where T represents the true triples set. To get rid of the noise in knowledge graph, we construct T with top 10% triples which fit the KGE the best. We discuss the detailed implementation of the generator and discriminator as follows. To identify the error in knowledge graph, first we have to achieve reliable examples of noisy data. Since such data is costly and labor intensive to obtain, we instead introduce a generator to produce them. The main goal of the generator is to generator high-quality fake triples that can fool discriminators, which aims at bringing a better classifier to identify error in knowledge graph. In addition, to simultaneously make KGE model aware of the existing noisy data, we also explicitly incorporate the noisy triples generated by the generator as negative samples to train the noise aware KGE model. Note that, the generator takes embedding vectors learned by the noise-aware KGE model as input. Given a true triples (h, r, t), the generator aims to select the most likely triples to be the noisy triples (h, r, t) from its negative samples candidate set N (h, r, t). To achieve this goal, a two-layer fully-connected neural network is introduced to model the probability distribution over the candidate negative samples N (h, r, t). In particular, this MLP uses ReLU as the activation function for the first layer and adds Softmax to the second layer. It takes the concatenation of the embedding vectors h, r and t of triple (h, r, t) as input and output the probability whether the triple (h, r, t) is the most likely noisy triple. As the output of the generator is a discrete triple, training for the generator is a little tricky due to data indifferentiability issue. A common solution is to use policy gradient based reinforcement learning instead (a). In particular, for a generated triple, the reward from discriminator is defined as f D (h, r, t), which is exactly the probability of the triple (h, r, t) to be true. Thus, in order to fool the discriminator, the generator is trained to maximize the expected reward as follows: At the very beginning, because of lack of guidance, the generator generates "noise" by random sampling. However, such "noise" is too easy for discriminator to distinguish and in a low reward. By learning to maximize the reward, the generator continually learns better probability distribution of the candidate negative samples and keep generating "more confusing" triples to improve the capacity of the discriminator. Moreover, this generator can also be used to produce high quality negative samples for the noise aware KGE model. Discriminator aims to distinguish the true triples from the noisy triples. Both positive and negative examples are required to train the discriminator. Generator learns to provide reliable negative examples for the discriminator while observed triples in knowledge graph can be regarded as positive examples. However, as knowledge graph is noisy, considering all observed triples as positive samples will inevitably introduce noise. We therefore utilize an interesting observation, when noisy data exist, deep learning models tend to memorize these noise in the end, which leads to the poor generalization performance . Specifically, we take top 10% triples which fit the KGE the best (with lowest f r (h, t)) as positive training examples. Considering discriminator is essentially a binary classifier, the objective function of the discriminator can be formulated as minimizing the following cross entropy loss: where f D (h, r, t) = σ(MLP(h, r, t)) where MLP is a two-layer neural network with ReLU as activation function and σ(x) = 1/(1 + exp(−x)) is the standard sigmoid function. If we use TransE as our KGE model, the MLP takes the vector h + r − t as input and output the probability of the triple (h, r, t) being true. f D (h, r, t) will later be used to define C(h, r, t) for each positive triple (h, r, t) in knowledge graph. To be specific, C(h, r, t) can be either a binary value or a soft value. When it is a binary variable, it represents the classification regard to whether a triple is a true triple. When it takes a soft value, it indicates the probability of the triple (h, r, t) being true. We denote the previous one as a hard version of our proposed model and the later one as the soft version. Require: Observed triples in knowledge bases T 1: Initialize C(h, r, t) as 1 for all triples. 2: Train noise aware KGE model with random negative sampling until convergence. 3: for n = 1: N do Take top 10% triples which fit the noise aware KGE the best as positive examples. Train GAN with the selected positive examples until convergence. 6: Update C(h, r, t) according to discriminator of GAN. Train noise aware KGE model with negative samples generated by the generator until convergence. 8 We start by training the noise aware KGE model. We initiate the confidence score C(h, r, t) as 1 for all triple facts and pretrain the noise aware KGE model with random negative sampling to gain reliable embedding vector. With these embedding vectors, we start the training of GAN model from the generator. The generator learns to understand the probability distribution of the candidate negative samples so as to select the most likely noisy data. In particular, the discriminator gives it a reward as guidance. Once the generator finishes training, it can be used to prepare noisy data for the discriminator. In addition, we take top 10% triples facts which fit the noise aware KGE the best as credible triple facts. With noisy data as negative examples and triple facts as positive examples, discriminator learns a classifier to distinguish the positive triples from the negative triples. After we obtain the well trained GAN model, given the confidence score assigned by the discriminator and the negative samples generated by the generator, we retrain the noise aware KGE model and update the embedding vectors. The process repeats until convergence. The embeddings learned by the noise aware KGE model is regarded as our final representation for entities and relations. Datasets. We evaluate our NoiGAN on five benchmark datasets, including FB15K, WN18,FB15K-237, WN18RR and YAGO3-10. These benchmark datasets benefit from human curation that in highly reliable facts. To simulate the real-world knowledge graphs extracted automatically from unstructured text data, we modified these benchmark datasets to include noisy triples. Since all kinds of noise might be contained while we construct knowledge graphs, our approach to introducing noise is to substitute the true head entity or tail entity with any randomly selected entity. Following this approach, we construct five KGs based on each benchmark dataset with noisy triples to be different ratio of (e.g., 40%, 70% and 100%) of true triples. All noisy datasets share the same entities, relations, validation and test sets with the original benchmark dataset, with all generated noisy triples fused into the original training set. The statistics of these knowledge graphs are summarized in Table 1. Baselines. NoiGAN is compared with the following state-of-the-art algorithm, including KGE models (e.g., TransE ,DistMult and RotateE ), robust KGE models (e.g., attention based method ), noise aware KGE models (e.g., CKRL ) and KGE models with GAN (e.g., KBGAN ). In particular, there are three kinds of triple confidences defined in CKRL . In our paper, we take CKRL with local triple confidence, called CKRL (LT), as baseline. To fairly compare different methods, the same loss function and negative sampling strategies are employed for all models. Experimental Setup of NoiGAN. We evaluate two versions of our NoiGAN model as mentioned in Section 3.2.2. The soft version is denoted as NoiGAN (soft) while the hard version is denoted as NoiGAN (hard). To show that our NoiGAN can be easily generalized to various KGE models, TransE and RotateE methods by a grid search strategy. The range of different parameters is set as follows: embedding dimension k ∈ {250, 500, 1000}, batch size b ∈ {256, 512, 1024}, and fixed margin γ ∈ {9, 12, 24}. Afterwards, we compare the best of different methods. Both the entity embeddings and the relation embeddings are uniformly initialized and no regularization is imposed on them. As mention in Section 3.2, we implement both discriminator and the generator as simple two-layer fully connected neural networks. The size of hidden states for each of the two networks is set to 10. To verify the capability of NoiGAN in distinguishing noises in knowledge graphs, we evaluate NoiGAN in terms of classification performance. To be specific, we classify the training data by determining whether a triple is being true. For NoiGAN (hard), we can directly utilize the discriminator to classify noise. Whereas, for NoiGAN (soft), the discriminator only assigns a soft score between 0 to 1 to each triple, indicating the probability of a triple being true. We thus classify the triples by regarding the triple who have C(h, r, t) > 0.5 as the true triple and the remaining as noise. The experiments are conducted on three benchmark datasets, including FB15K, WN18 and YAGO3-10 with noisy triples to be different ratio of 10%, 20% and 40% of true triples. Although FB15K and WN18RR suffer from test triple leakage in the training set , we report the over training data, which won't be affected by the issue. As we make similar observations on NoiGAN-TransE and NoiGAN-RotatE, we only report the experimental w.r.t. NoiGAN-TransE to save space. Evaluation Metric. Two classification evaluation metrics are adopted, including AUC; A special measurement defined as the proportion of actual noises that are correctly identified. It can be calculated as TN TN+FP, where TN represents true negative while FP represents false positive. Results. The w.r.t. NoiGAN-TransE can be found in Table 2. We compared against CKRL (LT) as it is the only baseline method which is able to detect noise in the training dataset. We can observe that: Our model consistently achieves the best performances over all datasets in all cases, which demonstrates the capability of our models in detecting noises. NoiGAN-TransE (hard) has significant improvements in noise detection compared to NoiGAN-TransE (soft). Considering that NoiGAN-TransE (soft) merely assigns low confidence scores to noisy triples while NoiGAN-TransE (hard) completely eliminates these data, this further validates that our method can classify noisy data accurately. The baselines tend to classify all triples into the same class, which shows their inability in detecting noise. In addition to its promising on noise detection, our approach is also superior to existing stateof-the-art algorithms in terms of the quality of learned embedding. We conduct experiments on three benchmark datasets, including FB15K-237, WN18RR and YAGO3-10 with noisy triples to be different ratio of 40%, 70% and 100% of true triples... Evaluation Metric. We mask the head or tail entity of each test triple, and require each method to predict the masked entity. During evaluation, we use the filtered setting . The Hit@K (H@K) and Mean Reciprocal Rank (MRR) are adopted as the evaluation metrics. Results. Results of reasoning are shown in Table 3. In particular, as attention based method exceed the memory capacity of our machines on YAGO3-10 dataset, only the on the FB15K-237 and WN18RR datasets are reported. Note that TransE, CKRL (LT) and NoiGAN-TransE share the same score function TransE, they are comparable with each other. Similarly, RotatE and NoiGANRotatE share the same score function RotatE and thus they are comparable with each other. We make the following observations: NoiGAN consistently outperforms the baseline methods which share the same score function with it on noisy dataset sets. The performance gain is significant especially on datasets with 100% noise. Either NoiGAN-TransE or NoiGAN-RotatE will achieve the best performance on almost all noisy datasets. In particular, hard version of NoiGAN performs better then soft version in most cases. Even though attention based method claims that they can ensure robust performance, the show that our NoiGAN significantly outperforms it in terms of robustness. If we do not introduce noise, the performance of NoiGAN models is almost the same as their variants. In addition, the improvement introduced by the NoiGAN becomes more significant as the noise rate in KGs rises. This further proves the robustness of the NoiGAN. To demonstrate the power of discriminator in distinguishing noisy triples, we present some triples and their confidence score in Table 4. We conduct experiments over FB15K with 40% noise using the NoiGAN-TransE (hard) model. All hyperparameters are set as described in Section 4.1. To be clear, distance of a triple is calculated as h + r − t 1 while the scores of the triples are learned by the discriminator. As shown in 4, we can see that TransE performs poorly at detecting some "triky" errors, such as logic error (e.g., (head teacher, /people/profession/people with this profession, Michael Dobson)) and grammar error (e.g. (McDonalds, /dining/restaurant/cuisine, Joe Walsh),). To our surprise, NoiGAN (hard) has the ability to detect both. In this paper, we propose a novel framework NoiGAN, to jointly combine the tasks of knowledge graph completion and error detection for noise aware knowledge graph embedding learning. It consists of two main components, a noise-aware KGE model for knowledge graph completion and an adversarial learning framework for error detection. Under the framework, the noise-aware KGE model and the adversarial learning framework can alternately and iteratively boost performance of each other. Extensive experiments show the superiority of our proposed NoiGAN both in regard to knowledge graph completion and error detection.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We proposed a unified Generative Adversarial Networks (GAN) framework to learn noise-aware knowledge graph embedding.

Represent the short summary for finding the corresponding scientific article (long)

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.