Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Recommendation is a prevalent application of machine learning that affects many users; therefore, it is crucial for recommender models to be accurate and interpretable. In this work, we propose a method to both interpret and augment the predictions of black-box recommender systems. In particular, we propose to extract feature interaction interpretations from a source recommender model and explicitly encode these interactions in a target recommender model, where both source and target models are black-boxes. By not assuming the structure of the recommender system, our approach can be used in general settings. In our experiments, we focus on a prominent use of machine learning recommendation: ad-click prediction. We found that our interaction interpretations are both informative and predictive, i.e., significantly outperforming existing recommender models. What's more, the same approach to interpreting interactions can provide new insights into domains even beyond recommendation. Despite their impact on users, state-of-the-art recommender systems are becoming increasingly inscrutable. For example, the models that predict if a user will click on an online advertisement are often based on function approximators that contain complex components in order to achieve optimal recommendation accuracy. The complex components come in the form of modules for better learning relationships among features, such as interactions between user and ad features (; ; ; ;). Although efforts have been made to understand the feature relationships, there is still no method that can interpret the feature interactions learned by a generic recommender system, nor is there a strong commercial incentive to do so. In this work, we identify and leverage feature interactions that represent how a recommender system generally behaves. We propose a novel approach, Global Interaction Detection and Encoding for Recommendation (GLIDER), which detects feature interactions that span globally across multiple data-instances from a source recommender model, then explicitly encodes the interactions in a target recommender model, both of which can be black-boxes. GLIDER achieves this by first utilizing feature interaction detection with a data-instance level interpretation method called LIME over a batch of data samples. GLIDER then explicitly encodes the collected global interactions into a target model via sparse feature crossing. In our experiments on ad-click recommendation, we found that the interpretations generated by GLIDER are informative, and the detected global interactions can significantly improve the target model's prediction performance, even in a setting where the source and target models are the same. Because our interaction interpretation method is very general, we also show that the interpretations are informative in domains outside of recommendation, such as image and text classification. Our contributions are as follows: 1. We propose GLIDER to detect and explicitly encode global feature interactions in blackbox recommender systems. Through experiments, we demonstrate the overall interpretability of detected feature interactions and show that they can be leveraged to improve recommendation accuracy. A simplified overview of GLIDER. 1 GLIDER uses interaction detection and LIME together to interpret feature interactions learned by a source black-box (recommender) model at a data instance, denoted by the large green plus sign. 2 GLIDER identifies interactions that consistently appear over multiple data samples, then explicitly encodes these interactions in a target black-box recommender model f rec. Interaction Interpretations: A variety of methods exist to detect feature interactions learned in specific models but not black-box models. For example, RuleFit , Additive Groves , and Tree-Shap detect interactions learned in trees, and Neural Interaction Detection detects interactions learned in a multilayer perceptron. Some methods have attempted to interpret feature groups in black-box models, such as Anchors , Agglomerative Contextual Decomposition , and Context-Aware methods ; however, these methods were not intended to identify feature interactions. Explicit Interaction Representation: There are increasingly methods for explicitly representing interactions in models. , , , and directly incorporate multiplicative cross terms in neural network architectures and use attention as an interaction module, all of which are intended to improve the neural network's function approximation. This line of work found that predictive performance can improve with dedicated interaction modeling. followed up by detecting interactions in data then explicitly encoding them via feature crossing. Our work approaches this problem from a model interpretation standpoint to show that interaction interpretations are also useful in explicit encoding. Black-Box Local vs. Global Interpretations: Data-instance level local interpretation methods are more flexible at explaining general black-box models; however, global interpretations, which cover multiple data instances, have become increasingly desirable to better summarize model behavior. Locally Interpretable Model-Agnostic Explanations (LIME) and Integrated Gradients are some of most used methods to locally interpret any classifier and neural predictor respectively. There are some methods for global black-box interpretations, such as shuffle-based feature importance , submodular pick , and visual concept extraction . §4.1 of this paper discusses local interaction interpretations, and §4.2-4.4 explains how we extract and utilize global interaction interpretations. Notations: Vectors are represented by boldface lowercase letters, such as x or w. The i-th entry of a vector x is denoted by x i. For a set S, its cardinality is denoted by |S|. Let d be the number of features in a dataset. An interaction, I, is the indices of a feature subset: I ⊆ {1, 2, . . ., d}, where interaction order |I| is always ≥ 2. A higher-order interaction always has order ≥ 3. For a vector x ∈ R d, let x I ∈ R |I| be restricted to the dimensions of x specified by I. Let a black-box model be f (·): R p → R. A black-box recommender model uses tabular feature types, as discussed later in this section. In classification tasks, we assume f is a class logit. p and d may be different depending on feature transformations. Feature Interactions: By definition, a model f learns a statistical (non-additive) feature interaction I if and only if f cannot be decomposed into a sum of |I| arbitrary subfunctions δ i, each not depending on a corresponding interaction variable (; ;), i.e., f (x) = i∈I δ i (x {1,...,d}\i ). For example, a multiplication between two features, x 1 and x 2, is a feature interaction because it cannot be represented as an addition of univariate functions, i.e., x 1 x 2 = δ 1 (x 2) + δ 2 (x 1). Recommendation Systems: A recommender system, f rec (·), is a model of two feature types: dense numerical features and sparse categorical features. Since the one-hot encoding of categorical feature x c can be high-dimensional, it is commonly represented in a low-dimensional embedding e c = one hot(x c)v c via embedding matrix v c. We now discuss the different components of GLIDER, starting from data-instance level (local) interpretations of interactions in §4.1, then global interaction detection in §4.2, and finally explicitly encoding the global interactions in §4.3. While our methodology is focused on recommender systems, it is not necessarily limited to this model type. Nonetheless, recommender systems are interesting because they have pervasive application in real-world systems, and their features are often very sparse. By sparse features, we mean features with many categories, e.g., millions of user IDs. The sparsity makes interaction detection challenging especially when applied directly on raw data because the one-hot encoding of sparse features creates an extremely large space of potential feature combinations . We start by explaining how to detect feature interactions in a black-box model at the data-instance level via interaction detection on feature perturbations. LIME Perturbation and Inference: Given a data instance x ∈ R p, LIME proposed to perturb the data instance by sampling a separate binary representation x ∈ {0, 1} d of the same data instance. Let ξ: {0, 1} d → R p be the map from the binary representation to the perturbed data instance. Starting from a binary vector of all ones that map to the original features values in the data instance, LIME uniformly samples the number of random features to switch to 0 or the "off" state. In the data instance, "off" could correspond to a 0 embedding vector for categorical features or mean value over a batch for numerical features. It is possible for d < p by grouping features in the data instance to correspond to single binary features in x. A key step in LIME interpretations is obtaining black-box predictions for the perturbed data instances to generate a dataset with binary inputs and prediction targets: Feature Interaction Detection: Feature interaction detection is concerned with identifying feature interactions in a dataset (; ; ;). Typically, proper interaction detection requires an expensive pre-processing step of feature selection to remove correlated features that adversely affect detection performance . Since the features in D are sampled randomly, they are uncorrelated by default, so we can directly use dataset D to detect feature interactions from black-box model f at a data instance x. f can be an arbitrary function and can generate highly nonlinear targets in D, so we focus on detecting interactions that could have generic forms. In light of this, we use a state-of-the-art method called Neural Interaction Detection (NID), which accurately and efficiently detects generic nonadditive and arbitrary-order statistical feature interactions . NID detects these interactions by training a lasso-regularized multilayer perceptron (MLP) on a dataset, then identifying the features that have high-magnitude weights to common hidden units. NID is efficient by greedily testing the top-interaction candidates of every order at each of h first-layer hidden units, enabling arbitrary-order interaction detection in O(hd) tests within one MLP. We can now define a function, MADEX(f, x), that inputs black-box f and data instance x, and outputs, a set of top-k detected feature interactions. MADEX stands for "ModelAgnostic Dependency Explainer". As the name suggests, MADEX is not limited to recommender models; it can also be used for general black-box models. In some cases, it is necessary to identify a k threshold. Because of the importance of speed for local interpretations, we simply use a linear regression with additional multiplicative terms to approximate global interactions I i and their counts c i over the dataset 1: G ← initialize occurrence dictionary for global interactions 2: for each data sample x within dataset B do 3: G ← increment the occurrence count of I j ∈ S, ∀j = 1, 2,..., |S| 5: sort G by most frequently occurring interactions 6: [optional] prune subset interactions in G within a target number of interactions K the gains given by interactions in S, where k starts at 0 and is incremented until the linear model's predictions stop improving. In this section, we discuss the first step of GLIDER. As defined in §4.1, MADEX takes as input a blackbox model f and data instance x. In the context of this section, MADEX inputs a source recommender system f rec and data instance x = [x 1, x 2, . . ., x p]. x i is the i-th feature field and is either a dense or sparse feature. p is both the total number of feature fields and the number of perturbation variables (p = d). We define global interaction detection as repeatedly running MADEX over a batch of data instances, then counting the occurrences of the same detected interactions, shown in Alg. 1. The occurrence counts are not only a useful way to rank global interaction detections, but also a sanity check to rule out the chance that the detected feature combinations are random selections. One potential concern with Alg. 1 is that it could be slow depending on the speed of MADEX. In our experiments, the entire process took less than 3 hours when run in serial over a batch of 1000 samples with ∼ 40 features on a 32-CPU server. In addition, this algorithm is fully parallelizable and only needs to be run once to obtain the summary of global interactions. Each global interaction I i from Alg. 1 is used to create a synthetic feature x Ii for a target recommender system. The synthetic feature x Ii is created by explicitly crossing sparse features indexed in I i. If interaction I i involves dense features, we bucketize the dense features before crossing them. The synthetic feature is sometimes called a cross feature or conjunction feature . In this context, a cross feature is an n-ary Cartesian product among n sparse features. If we denote X 1, X 2,..., X n as the set of IDs for each respective feature x 1, x 2,..., x n, then their cross feature x {1,...,n} takes on all possible values in Accordingly, the cardinality of this cross feature is |X 1 | × · · · × |X n | and can be extremely large, yet many combinations of values in the cross feature are likely unseen in the training data. Therefore, we generate a truncated form of the cross feature with only seen combinations of values, x (j) I, where j is a sample index in the training data, and x (j) I is represented as a sparse ID in the cross feature x I. We further reduce the cardinality by requiring the same cross feature ID to occur more than T times in a batch of samples, or set to a default ID otherwise. These truncation steps significantly reduce the embedding sizes of each cross feature while maintaining their representation power. Once cross features {x Ii} i are included in a target recommender system, it can be trained as per usual. There are dual perspectives of GLIDER: as a method for model distillation or model enhancement. If a strong source model is used to detect global interactions which are then encoded in more resourceconstrained target models, then GLIDER adopts a teacher-student type distillation process. If interaction encoding augments the same model where the interactions were detected from, then GLIDER tries to enhance the model's ability to represent the interactions. In our experiments, we study the effectiveness of GLIDER on real-world data. The hyperparameters for local interaction interpretation in our experiments are as follows. For all experiments, we use 5000 perturbation samples to train the models used for interaction detection. We use NID as the interaction detector, which requires training an MLP to detect each set of interactions. The MLPs for §5.3 have architectures of 50-30-10 first-to-last hidden layer sizes, and in §5.2, architectures of 256-128-64. We apply an 1 regularization of λ 1 = 5e−5, and the learning rate is 5e−3. In general, models are trained with early stopping on the validation set. For LIME perturbations, we need to establish what a binary 0 maps to via ξ in the raw data instance (§4.1). In domains involving embeddings, i.e., sparse features and word embeddings, the 0 ("off") state is the zeroed embedding vector. For dense features, it is the mean feature value over a batch; for images, the mean of each RGB of the image. For our DNA experiment, we use a random nucleotide other than the original one. These settings correspond to what is used in literature (; . In our graph experiment, the nodes within the neighborhood of a test node are perturbed, where each node is zeroed during perturbation. In this section, we provide experiments with GLIDER on models trained for clickthrough-rate (CTR) prediction. The recommender models we study include commonly reported baselines, which all use neural networks: Wide&Deep , DeepFM , Deep&Cross , xDeepFM , and AutoInt . AutoInt is the reported state-of-the-art in academic literature, so we use the model settings and data splits provided by AutoInt's official public repository 1. For all other recommender models, we use public implementations 2 with the same original architectures reported in literature, set all embedding sizes to 16, and tune the learning rate and optimizer to try to reach or surpass the test logloss reported by the AutoInt paper (on AutoInt's data splits). From tuning, we use the Adagrad optimizer with learning rates in {0.01, 0.001}. The datasets we use are benchmark CTR datasets with the largest number of features: Criteo 3 and Avazu 4, whose data statistics are shown in Table 1. Criteo and Avazu both contain 40+ millions of user records on clicking ads, with Criteo being the primary benchmark in CTR research (Cheng Table 3 : Test prediction performance by encoding top-K global interactions in baseline recommender systems on the Criteo and Avazu datasets (5 trials). K are 40 and 20 for Criteo and Avazu respectively. "+ GLIDER" means the inclusion of detected global interactions to corresponding baselines. The "Setting" column is labeled relative to the source of detected interactions: AutoInt.;;;; ). For each dataset, we train a source AutoInt model, f rec, then run global interaction detection via Algorithm 1 on a batch of 1000 samples from the validation set. A full global detection experiment finishes between 2-3 hours when run in serial on either Criteo or Avazu datasets in a 32-CPU Intel Xeon E5-2640 v2 @ 2.00GHz server, and significant speed-ups can be achieved by fully parallelizing Algorithm 1. The detection across datasets are shown in Figure 2 as plots of detection counts versus rank. Because the Avazu dataset contains non-anonymized features, we directly show its top-10 detected global interactions in Table 2. From Figures 2, we see that the top interactions are detected very frequently across data instances, once appearing across more than half of the batch. In Table 2, the top-interactions can be explained. For example, the interaction between "hour" (in UTC time) and "device ip" makes sense because users -here identified by an IP address -have ad-click behaviors dependent on their time zones. We hypothesize that the global interaction detections are also informative for modeling purposes. Based on our from the previous section (§5.2.1), we turn our attention to explicitly encoding the detected global interactions in target baseline models via truncated feature crosses (detailed in §4.3). In order to generate valid cross feature IDs, we bucketize dense features into a maximum of 100 bins before crossing them and require that final cross feature IDs occur more than T = 100 times over a training batch of one million samples. We take AutoInt's top-K global interactions on each dataset from §5.2.1 with subset interactions excluded (Algorithm 1, line 6) and encode the interactions in each baseline model including AutoInt Table 5: Prediction performance (mean-squared error; lower is better) with (k > 0) and without (k = 0) interactions for random data instances in the test sets of respective black-box models. k = L corresponds to the interaction at a rank threshold. 2 ≤ k < L are excluded because not all instances have 2 or more interactions. Only with detected interactions are shown. At least 80% (≥ 320) of the data instances possessed interactions over 10 trials for each model/performance statistic. linear LIME 0 9.8e−3 ± 9e−4 0.101 ± 7e−3 0.25 ± 0.07 0.080 ± 3.0e−4 MADEX 1 8e−3 ± 1e−3 0.056 ± 9e−3 0.22 ± 0.06 0.062 ± 8.1e−3 MADEX L 6e−3 ± 1e−3 0.024 ± 7e−3 0.16 ± 0.05 0.038 ± 9.6e−3 itself. There is consensus that 0.001 logloss or AUC improvements are significant in CTR prediction tasks (; ; ;). K is tuned on valiation sets, and model hyperparameters are the same between a baseline and one with encoded interactions. We set K = 40 for Criteo and K = 20 for Avazu. We found that using GLIDER can often reach or exceed the 0.001 significance level, especially for the main Criteo benchmark dataset, as shown in Table 3. These performance gains are obtained at limited cost of extra model parameters (Table 4) thanks to the truncations applied to our feature crosses. In Figure 3, we also show how the test performance of AutoInt varies with different K on the Criteo dataset. One one hand, the evidence that AutoInt's detected interactions can improve other baselines' performance suggests the viability of interaction distillation. On the other hand, evidence that AutoInt's performance on Criteo can improve using its own detected interactions suggests that AutoInt may benefit from learning interactions more explicitly. In either model distillation or enhancement settings, we found that GLIDER performs especially well on industry production models trained on large private datasets with thousands of features. Since the proposed interaction interpretations by GLIDER are not entirely limited to recommender systems, we demonstrate interpretations on more general black-box models. Specifically, we experiment with the function MADEX(·) defined in §4.1, which inputs a black-box f, data-instance x, and outputs a set of interactions. The models we use are trained on very different tasks: ResNet152-an image classifier pretrained on ImageNet'14 , Sentiment-LSTM-a 2-layer bi-directional long short-term memory network (LSTM) trained on the Stanford Sentiment Treebank (SST) , DNA-CNN-a 2-layer 1D convolutional neural network (CNN) trained on MYC-DNA binding data (; ; ; ;), and GCN-a 3-layer Graph Convolutional Network trained on the Cora dataset . We first provide quantitative validation for the detected interactions of all four models in §5.3.1, followed by qualitative for ResNet152, Sentiment-LSTM, and DNA-CNN in §5.3.2. To provide quantitative validation of interaction interpretations of black-box models, we evaluate the predictive power of the interactions at the data instance level. As suggested in §4.1 and §4.3, encoding feature interactions is a way to increase a model's function representation, but this also means that prediction performance gains over simpler first-order models (e.g., linear regression) is a way to test the significance of the detected interactions. In this section, we use neural network function approximators for each top-interaction from the ranking {I i} given by MADEX's interaction detector (in this case NID). Similar to the k-thresholding description in §4.1, we start at k = 0, which is a linear regression, then increment k with added MLPs for each until validation performance stops improving, denoted at k = L. The MLPs all have architectures of 30-10 first-to-last hidden layer sizes and use the binary perturbation dataset D (introduced in §4.1). Test prediction performances are shown in Table 5 for k ∈ {0, 1, L}. The average number of features of D among the black-box models ranges from 16 to 189. Our quantitative validation shows that adding feature interactions for DNA-CNN, Sentiment-LSTM, and ResNet152, and adding node in-teractions for GCN in significant performance gains when averaged over 40 randomly selected data instances in the test set. Figure 4: Qualitative of the detected interactions by MADEX and the selected features by LIME's original linear regression ("LIME selection") on (a) ResNet152 and (b) Sentiment-LSTM. The interpretations between MADEX and LIME selection are complementary. For our qualitative analysis, we provide interaction interpretations via MADEX(·) of ResNet152, Sentiment-LSTM, and DNA-CNN on test examples. The interpreations are given by, a set of k detected interactions, which are shown in Figures 4a and 4b for ResNet152 and Sentiment-LSTM respectively. Interactions that have majority overlap among S are merged, i.e., overlap coefficient > 0.5 . For reference, we also show the selected features by LIME's original linear regression, which takes the top-5 features that attribute towards the predicted class 5. In Figure 4a, the MADEX columns show selected features from the detected interactions between Quickshift superpixels . We see that the interactions can form a single region or multiple regions of the image, and they are complementary to LIME's feature selection. For example, the interpretations of the "deskop computer" classification show that interaction detection finds one of the computers and feature selection finds the other. For Sentiment-LSTM interpretations in Figure 4b, we also see that MADEX's interactions can complement LIME's selected features. Here, the interactions show salient combinations of words, such as "science fiction" and "I like pug". In our experiments on DNA-CNN, we consistently detected the interaction between "CACGTG" nucleotides, which form a canonical DNA sequence . The interaction was detected 76.5% out of 187 CACGTG appearances in the test set. We proposed GLIDER that detects and explicitly encodes global feature interactions in black-box recommender systems. In our experiments, we found that the detected global interactions are informative and that explicitly encoding interactions can improve the accuracy of CTR predictions. We further validated interaction interpretations on image, text, and graph classifiers. We hope GLIDER encourages investigation into the complex interaction behaviors of recommender models to understand why certain feature interactions are very predictive. For future research, we wish to understand how feature interactions play a role in the integrity of automatic recommendations. In this section, we study whether increasing embedding size can obtain similar prediction performance gains as explicitly encoding interactions via GLIDER. We increase the embedding dimension sizes of every sparse feature in baseline recommender models to match the total number of model parameters of baseline + GLIDER as close as possible. The embedding sizes we used to obtain similar parameter counts are shown in Table 6. For the Avazu dataset, most of the embedding sizes remain unchanged because they were already the target size. The corresponding prediction performances of all models are shown in Table 7. We observed that directly increasing embedding size / parameter counts generally did not give the same level of performance gains that GLIDER provided. Table 6: Comparison of # model parameters between baseline models with enlarged embeddings and original baselines + GLIDER (from Tables 3 and 4). The models with enlarged embeddings are denoted by the asterick (*). The embedding dimension of sparse features is denoted by "emb. size". Percent differences are relative to baseline* models. M denotes million, and the ditto mark (") means no change in the above line. Table 7: Test prediction performance coresponding to the models shown in Table 6 Model Criteo Avazu We examine the effect of dense feature bucketization on the parameter efficiency and prediction performance of AutoInt. Results are provided for the Criteo dataset, which contains 13 dense features. Figure 5 shows the effects of varying the number of dense feature buckets on the total number of parameters and the test logloss of AutoInt. Figure 6 shows the effects of varying the number of dense buckets on the embedding sizes of the cross features involving dense features. Both the effects on the average and individual embedding size are shown. 20 of the cross features involved a dense feature. Patterns to note include the largely asymptotic behavior of the parameter plots as the number of buckets increases (Figures 5a and 6). Our requirement that a valid cross feature ID occurs more than T times (§4.3) restricts the growth in parameters. In some cases, the number of cross feature IDs (embedding size) decreases (Figure 6b), which happens when the dense bucket size becomes too small to satisfy the T occurrence restriction. In Figure 5b, prediction performance degrades beyond 100 buckets, yet it is still an improvement over the baseline without cross features (0.4434 in Table 3). The degradation may be caused by overfitting. In this section, we show ranked for interactions discovered by MADEX in Sentiment-LSTM. The top-1 interactions are provided on random sentences from the SST test set. For every sentence, we preprocess it to remove stop words. If interactions are not detected in a sentence, that sentence is excluded. Results are shown in Table 8. We use the same stop words suggested by , i.e., {a, an, and, are, as, at, be, by, for, from, has, he, in, is, it, its, of, on, that, the, to, was, were, will, with}. In order to obtain occurrence counts of word interactions, we need to detect the same word interactions across different sentences. Naturally, different sentences will often have different words, so it is nontrivial to identify consistent word interactions. We start by collecting interaction candidates by running MADEX over all sentences in the SST test set, then identifying the word interactions that appear multiple times. Here, we make the assumption that word interactions are ordered but not necessarily adjacent or positionally bound. Therefore, a hypothetical interaction, (not, good), could be found in either sentence: "This is not good", or "This movie is not so good". The order of the words matter, so (not, good) = (good, not). After collecting interaction candidates, we then look for sentences in the larger IMDB dataset that contain the same ordered words of an interaction candidate. Therefore, each interaction candidate I i will have its own set of sentences W i that could potentially yield that interaction. Let W i contain a random selection of 40 viable sentences for I i. The interactions {I i} with the highest detection counts across corresponding W i are shown in We compare the detection performances between MADEX and baselines on identifying feature interactions learned by complex models, i.e., XGBoost , Multilayer Perceptron (MLP), and Long Short-Term Memory Network (LSTM) . The baselines are Tree-Shap: a method to identify interactions in tree-based models like XGBoost , MLP-ACD+: a modified version of ACD to search all pairs of features in MLP to find the best interaction candidate, and LSTM-ACD+: the same as MLP-ACD+ but for LSTMs. All baselines are local interpretation methods. For MADEX, we sample continuous features from a truncated normal distribution N (x, σ 2 I) centered at a specified data instance x and truncated at σ. We evaluate interaction detection performance by using synthetic data where ground truth interactions are known . We generate 10e3 samples of synthetic data using functions F 1 − F 4 (Table 10) with continuous features uniformly distributed between −1 to 1. Next, we train complex models (XGBoost, MLP, and LSTM) on this data. Lastly, we run MADEX and the baselines on 10 trials of 20 data instances at randomly sampled locations on the synthetic function domain. Between trials, the complex models are trained with different random initialization to test the stability of each interpretation method. Interaction detection performance is computed by the average R-precision 6 of interaction rankings across the sampled data instances. Results are shown in Table 11. On the tree-based model, MADEX can compete with the tree-specific baseline Tree-Shap, which only detects pairwise interactions. On MLP and LSTM, MADEX performs significantly better than ACD+. The performance gain is especially large in the LSTM setting. Table 11: Detection Performance in R-Precision (higher the better). σ = 0.6 (max: 3.2). "Tree" is XGBoost. *Does not detect higher-order interactions. †Requires an exhaustive search of all feature combinations. Tree-Shap MADEX MLP-ACD+ MADEX LSTM-ACD+ MADEX F 1 (x) 1 ± 0 1 ± 0 0.63 ± 0.08 1 ± 0 0.3 ± 0.2 1 ± 0 F 2 (x) 1 ± 0 0.14 ± 0.09 0.41 ± 0.06 0.97 ± 0.03 0.01 ± 0.02 0.96 ± 0.03 F 3 (x) 1 ± 0 1 ± 0 0.3 ± 0.2 1 ± 0 0.05 ± 0.08 1 ± 0 F 4 (x) * 0.2 ± 0.1 † 0.61 ± 0.07 † 0.54 ± 0.09 This section shows how often higher-order interactions are identified by GLIDER / MADEX. Figure 8 plots the occurrence counts of global interactions detected in AutoInt for the Criteo and Avazu dataset, which correspond to the in Figure 2. Here we only show the occurrence counts of higher-order interactions, where the exact interaction order is annotated besides each data point. 3rd-order interactions are the most common type, and interestingly, an 8th-order interaction appears 13 times in the Avazu dataset. Figure 9 plots histograms of interaction orders for all interactions detected from ResNet152 and Sentiment-LSTM across 1000 random samples in their test sets. The average number of features are 67 and 19 for ResNet152 and Sentiment-LSTM respectively. Higherorder interactions are especially common in ResNet152.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

Proposed a method to extract and leverage interpretations of feature interactions

Represent the TLDR for finding the corresponding scientific article.


Question: Represent the article for finding a single-sentence TLDR

Question: Rectified linear units, or ReLUs, have become a preferred activation function for artificial neural networks. In this paper we consider the problem of learning a generative model in the presence of nonlinearity (modeled by the ReLU functions). Given a set of signal vectors $\mathbf{y}^i \in \mathbb{R}^d, i =1, 2, \dots, n$, we aim to learn the network parameters, i.e., the $d\times k$ matrix $A$, under the model $\mathbf{y}^i = \mathrm{ReLU}(A\mathbf{c}^i +\mathbf{b})$, where $\mathbf{b}\in \mathbb{R}^d$ is a random bias vector, and {$\mathbf{c}^i \in \mathbb{R}^k$ are arbitrary unknown latent vectors}. We show that it is possible to recover the column space of $A$ within an error of $O(d)$ (in Frobenius norm) under certain conditions on the distribution of $\mathbf{b}$. Rectified Linear Unit (ReLU) is a basic nonlinear function defined to be ReLU: R → R + ∪ {0} as ReLU(x) ≡ max(0, x). For any matrix X, ReLU(X) denotes the matrix obtained by applying the ReLU function on each of the coordinates of the matrix X. ReLUs are building blocks of many nonlinear data-fitting problems based on deep neural networks (see, e.g., for a good exposition). In particular, showed that supervised training of very deep neural networks is much faster if the hidden layers are composed of ReLUs. Let Y ⊂ R d be a collection of signal vectors that are of interest to us. Depending on the application at hand, the signal vectors, i.e., the constituents of Y, may range from images, speech signals, network access patterns to user-item rating vectors and so on. We assume that the signal vectors satisfy a generative model, where each signal vector can be approximated by a map g: R k → R d from the latent space to the ambient space, i.e., for each y ∈ Y, y ≈ g(c) for some c ∈ R k. In this paper we consider the following specific model (single layer ReLU-network), with the weight (generator) matrix A ∈ R d×k and bias b ∈ R d: The generative model in raises multiple interesting questions that play fundamental role in understanding the underlying data and designing systems and algorithms for information processing. Here, we consider the following network parameter learning problem under the specific generative model of. Learning the network parameters: Given the n observations {y i} i∈[n] ⊂ R d from the model (cf.), recover the parameters of the model, i.e., A ∈ R d×k such that with latent vectors {c i} i∈[n] ⊂ R k. We assume that the bias vector b is a random vector comprising of i.i.d. coordinates with each coordinate distributed according to the probability density function 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. p(·). This question is closely related to the dictionary-learning problem. We also note that this question is different from the usual task of training a model (such as, ), in which case the set {c i} i∈[n] is also known (and possibly chosen accordingly) in addition to {y i} i∈[n]. Related works. There have been a recent surge of interest in learning ReLUs, and the above question is of basic interest even for a single-layer network (i.e., nonlinearity comprising of a single ReLU function). It is conceivable that understanding the behavior of a single-layer network would allow one to use some iterative peeling off technique to develop a theory for the generative models comprising of multiple layers. To the best of our knowledge, the network parameter learning problem, even for single-layer networks has not been studied as such, i.e., theoretical guarantees do not exist. Only in a very recent paper the unsupervised problem was studied when the latent vectors {c i} i∈ [n] are random Gaussian. The principled approaches to solve this unsupervised problem in practice reduce this to the'training' problem, such as the autoencoders that learn features by extensive end-to-end training of encoderdecoder pairs; or use the recently popular generative adversarial networks (GAN) that utilize a discriminator network to tune the generative network. The method that we are going to propose here can be seen as an alternative to using GANs for this purpose, and can be seen as an isolated'decoder' learning of the autoencoder. Note that the problem bears some similarity with matrix completion problems, a fact we greatly exploit. In matrix completion, a matrix M is visible only partially, and the task is to recover the unknown entries by exploiting some prior knowledge about M. In the case of, we are more likely to observe the positive entries of the matrix M, which, unlike a majority of matrix completion literature, creates the dependence between M and the sampling procedure.. With this notion, we can concisely represent the n observation vectors as where M = AC and 1 ∈ R n denotes the all-ones vector and ⊗ denotes the Kronecker product. Recall our assumption that b is a random vector comprising of i.i.d. coordinates with each coordinate distributed according to the probability density function p(·). Note that this model ensures that the bias corresponding to each coordinate is random, but does not change over different signal vectors. We employ a natural approach to learn the underlying weight matrix A from the observation matrix Y. As the network maps a lower dimensional latent vector c ∈ R k to obtain a signal vector y = ReLU(Ac + b) in dimension d > k, the matrix M = AC (cf.) is a low-rank matrix as long as k < min{d, n}. In our quest of recovering the weight matrix A, we first focus on estimating the matrix M, when given access to Y. This task can be viewed as estimating a low-rank matrix from its partial (randomized) observations. One of the main challenges that we face here is that while an entry of the matrix Y is a random variable (since b is a random bias), whether that is being observed or being cut-off by the ReLU function (for being negative) depends on the value of the entry itself. In general matrix completion literature, the entries of the matrix being observed are sampled independent of the underlying matrix itself (see, e.g., and references therein). For this reason, we cannot use most of these off-the-shelf. However, similar predicament is (partially) present in, where entries are quantized while being observed. This motivates us to employ a maximum-likelihood method inspired by. That said, our observation model differs from in a critical way: in our case the bias vector, while random, does not change over observations. This translates to less freedom during the transformation of the original matrix to the observed matrix, leading to dependence among the elements in a row. Furthermore, the analysis becomes notably different since the positive observations are not quantized. Furthermore, although our formulation is close to, because of the aforementioned differences in the observation models, we get much stronger guarantee on the recovery of the matrix. Indeed, our are comparable to analogous of that also study the quantized matrix completion problem. We show that our method guarantees the recovery of the matrix AC from Y with an error in Frobenius norm at most O(√ d) with high probability (see Theorem 1 for the formal statement). Then leveraging the well known on matrix perturbation, it is possible to also recover the column space of A with a similar guarantee. Extension to multi-layer networks. Our aim is to use a'peeling' technique to extend our to multi-layer networks. While rigorous theoretical guarantees for the network parameter learning problem do not extend to multi-layer networks as of now, we can still use the peel-off decoding for this case. Note that, even for the'training' problem of such network, which is a less demanding task than our problem, no rigorous guarantee exists beyond only two-layers. For example, the state-of-the art theoretical for training such as in, hold only for one layer of nonlinearity (the second layer is assumed to have a linear activation function in ). In fact most recent theoretical guarantees in this domain are restricted to one or two layer networks e.g.,. Given this, obtaining provable guarantees for multi-layer case of network parameter learning problem seems to be a challenging future work. We now focus on the task of estimating M from the observation matrix Y (cf. where M i,j denotes the (i, j)-th entry of M, and For i ∈ [d] and j ∈ [n], let M i,(j) denote the j-th largest element of the i-th row of M, i.e., for It is straightforward to verify from. Similarly, it follows from that whenever we have ). Based on these observation, we define the set of matrices X Y,ν,γ ⊂ R d×n as Recall that, p: R → R denotes the probability density function of each bias RV B. We use the notation F (x 1, x 2) = P(−x 1 ≤ B ≤ −x 2) and X * i = max j∈[n] X i,j. Thus, the (normalized) log-likelihood of observing Y given that X is the original matrix takes the following form. In order to recover the matrix M from the observation matrix Y, we employ the following program. Define ω p,γ,ν to be such that F (x, y) ≥ ω p,γ,ν for all x, y ∈ [−γ, γ] with |x − y| > ν. In what follows, we simply refer this quantity as ω p given that γ and ν are clear from context. Further define the following flatness and Lipschitz parameters associated with a function f: R → R: The following characterizes the performance of the program proposed in. Theorem 1. Assume that M ∞ ≤ γ and the observation matrix Y is related to M according to. Let M be the solution of the program specified in, and the bias density function p(x) is differentiable with bounded derivative. Then, the following holds with probability at least 1 − where, C 0 is a constant. The quantities β γ (p) and L γ (p) depend on the distribution of the bias and are defined in and, respectively. The full proof of Theorem 1 is omitted due to page limit. In the first step of the proof, we show that given the observation matrix Y, for any X ∈ X Y,ν,γ (cf.), we have This fact follows from the regularity assumptions on the probability density p and a sequence of inequalities similar to ones that relate KL divergence and the Hellinger divergence. In the next step, we show that, To upper bound the right-hand side we resort to standard techniques to bound the supremum of an empirical process such as the symmetrization and the contraction principle. The detail can be found in the full version. Recovering the network parameters. Let us denote the recovered matrix M as M = M + E, where E is the perturbation matrix that has bounded Frobenius norm (cf.). Now the task of recovering the parameters of the single-layer ReLU-network is equivalent to solving for A given M = M + E = AC + E. Note that, even without the perturbation E we could only hope to recover the column space of A and not the exact matrix A. Let U k and U k be the top k left singular vectors of M and M, respectively. Let σ k, the smallest non-zero singular value of M, is at least δ > 0. Then, it follows from standard from matrix-perturbation theory (cf. ) that there exists an orthogonal matrix O ∈ R k×k such that which is a guarantee that the column space of A is recovered by SVD within an error of O(d) in Frobenius norm by the column space of U k. Note that σ 1 is the largest singular value of M. Future directions: Extension to multi-layer networks. To learn a multi-layer ReLU network, we propose a'peeling' decoder as defined below. For better understanding, let us consider a two-layer model as follows: where Our'peeling' decoder will approach this layer-by-layer. First we use the likelihood-based matrix completion technique outlined in to recover M 2 as in the case of a one-layer network. Note that we will not be able to recover M 2 exactly, as Theorem 1 guarantees recovery only up to a Frobenius norm error. This creates a hurdle to recover A 1 C, since our current method does not handle dense bounded norm noise while learning network parameters. Also it will not be realistic to assume a probabilistic model for this bounded noise, since in the peeling off process the noise is coming from decoding of the previous layer. Under certain coherence conditions on the matrix to be learned, it is possible to handle such situations (see e.g., ). Note that, such coherence condition must be satisfied by the parameters of every layer of the network. An amenable but practical algorithm that is resilient to dense noise, under reasonable assumption on the network parameters, is an immediate area of interest. Even when this first hurdle can be crossed, we still have a task of factorization of M 2 to find A 2 and ReLU A 1 C + b 1 ⊗ 1 T, and in general we cannot find a unique factorization. Here as well, with additional reasonable assumptions the factorization can be made unique. Note that this might be simpler that the factorization step of the one-layer network, since there is already a lot of structure in the latter matrix (such as nonnegativity due to being output of a ReLU and about ∼ 50% sparsity). Provided this can be made to work analytically for two layers, there should not be any theoretical issue left to extend the process to multiple layers.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show that it is possible to recover the parameters of a 1-layer ReLU generative model from looking at samples generated by it

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.


Question: Represent the scientific article to find a short TLDR

Question: Methods that calculate dense vector representations for features in unstructured data—such as words in a document—have proven to be very successful for knowledge representation. We study how to estimate dense representations when multiple feature types exist within a dataset for supervised learning where explicit labels are available, as well as for unsupervised learning where there are no labels. Feat2Vec calculates embeddings for data with multiple feature types enforcing that all different feature types exist in a common space. In the supervised case, we show that our method has advantages over recently proposed methods; such as enabling higher prediction accuracy, and providing a way to avoid the cold-start problem. In the unsupervised case, our experiments suggest that Feat2Vec significantly outperforms existing algorithms that do not leverage the structure of the data. We believe that we are the first to propose a method for learning unsuper vised embeddings that leverage the structure of multiple feature types. Informally, in machine learning a dense representation, or embedding of a vector x ∈ R n is another vector y ∈ R r that has much lower dimensionality (r n) than the original representation, and can be used to replace the original vector in downstream prediction tasks. Embeddings have multiple advantages, as they enable more efficient training BID17, and unsupervised learning BID25. For example, when applied to text, semantically similar words are mapped to nearby points. We consider two kind of algorithms that use embeddings: Word2Vec BID17, are designed to provide embeddings that are useful for a wide-array of predictions tasks. For example, the loss function of the continuous bag of words (CBOW) algorithm of Word2Vec is tuned to predict the next word of a sequence; however, in practice, the embeddings produced are mostly used for other tasks, such as analogy solving BID17, or sentiment analysis BID15. In the context of this paper, we refer to the embeddings of an unsupervised method that can be used for a variety of auxiliary prediction tasks as general-purpose. 2. Supervised methods, like matrix factorization, produce embeddings that are highly tuned to a prediction task. These embeddings may be interpretable but do not usually generalize to other tasks. We refer to these embeddings as task-specific. Matrix factorization and Word2Vec are unable to calculate embeddings for items that are not available during training ("cold-start" problem). While recent work using n-gram features BID2 have addressed this limitation for supervised and unsupervised tasks, it can only be used for a single feature type-words. In this paper we propose Feat2Vec as a novel method that allows calculating embeddings of arbitrary feature types from both supervised and unsupervised data. Our main contributions are:• Unsupervised Feat2Vec. Existing general-purpose dense representation methods are largely restricted to one or two feature types. For example, the Word2Vec methods can only calculate embeddings for words, while follow-up work has enabled embeddings for both words and documents BID15. To our knowledge, Feat2Vec is the first algorithm that is able to calculate general-purpose embeddings that are not tuned for a single specific prediction task for arbitrary feature types.• Supervised Feat2Vec. Task-specific methods can use arbitrary feature types, but are restricted in that embeddings must be calculated for each individual feature, while sometimes higher-level of abstractions may be desirable-for example, we may want to have embeddings of documents instead of simply words. This capability makes Supervised Feat2Vec extremely flexible. We demonstrate that our method can be used to calculate embeddings of unseen (cold-start) items when there is an alternative textual description. Factorization Machine BID22 ) is one of the most successful methods for general-purpose factorization. BID22 formulated it as an extension to polynomial regression. Consider a degree-2 polynomial (quadratic) regression, where we want to predict a target variable y from a vector of inputs x ∈ R n: DISPLAYFORM0 In words, n is the total number of features, the term b 0 is an intercept, b i is the strength of the i-th feature, and w i,j is the interaction coefficient between the i-th and j-th feature. The function ω is an activation. Choices for ω include a linear link (ω(x) = x) for continuous outputs, or a logistic link (ω(x) =exp (x) exp(x)+1 ) for binary outputs. Factorization Machine replaces the two-way individual pairwise parameters w i,j for each interaction with a vector of parameters w i for each feature. This is a rank-r vector of latent factors-embeddings in the neural literature-that encode the interaction between features and replaces the quadratic regression model with the following: DISPLAYFORM1 Intuitively, the dot product (·) returns a scalar that measures the (dis)similarity between the latent factors of features x i and x j. Polynomial regression has n 2 interaction parameters, and Factorization Machine has n×r. While setting r n makes the model less expressive, factorization will typically exploit features having some shared latent structure. Factorization Machine may dramatically reduce the number of parameters to estimate. BID22 shows that when the feature vector x consists only of two categorical features in one-hot encoding, Factorization Machine is equivalent to the popular Matrix Factorization algorithm BID14 ). We now describe how Feat2Vec extends the Factorization Machine model by allowing grouping of features, and enabling arbitrary feature extraction functions (§ 3.1). We also report a supervised method to learning Feat2Vec (§ 3.2), as well as a novel unsupervised training procedure (§ 3.3). We propose a framework for extending factorization machine with neural methods, by introducing structure into the feature interactions. Specifically, we do this by defining feature groups, κ, where each group contains features of a particular type. Explicitly, κ is a partition of the set of feature columns in a dataset and each set within the partition is a feature group. The embeddings of a feature group are then learned via a feature extraction function, φ i, defined for each feature group. Feat2Vec will then extract features from each feature group, and build r latent factors from them. In Factorization Machine, all the feature embeddings interact with each other, while in Feat2Vec, the interactions only occur between different feature groups. Formally, the addition of deep extraction methods yields the following statistical model: DISPLAYFORM0 In this notation, x κi is a subvector that contains all of the features that belong to the group κ i. Thus, DISPLAYFORM1 The intuition is that by grouping (sub-)features as a single entity, we can can reason on a higher level of abstraction. Instead of individual sub-features interacting among each other, the embeddings of feature groups interact with those of other groups. φ i is a feature extraction that inputs the i-th feature group of the instance, and returns an r-dimensional embedding. The feature extraction function φ i can allow for an arbitrary processing of its subfeatures. Across groups, entities interact with each other via the output of φ only. As a concrete example of an application of this grouping/feature extraction, we might group the individual words of a document into a "document" feature group, and allow this document embedding to then interact with learned embeddings of other document metadata (such as author id). We might expect the extraction function φ for the words in a document to extract features that characterize the attributes of the document taken as a whole, rather than simply the sum of its individual words. Figure 1 compares existing factorization methods with our novel model. In this example, Feat2Vec is using two feature groups: the first group only has a single feature which is projected to an embedding (just like a regular Factorization Machine); the second group has multiple features, which are together projected to a single embedding. The simplest implementation for φ i is a linear fully-connected layer, where the output of the r-th entry is: DISPLAYFORM2 DISPLAYFORM3 Note that without loss of generality, we could define a model that is equivalent to a shallow Factorization Machine by allowing each feature group to be a singleton: κ = {{x 1}, {x 2}... {x n}} and the linear extraction function presented in Equation 4.We can use Feat2Vec to both use large feature sets and overcome the cold-start problem. This is only possible when there is an alternative description of the item available (for example an image or a passage of text). In FIG1, we show how we address this problem by treating the words as indexed features, but placed within a structured feature group κ w, the group of word features. A feature extraction function φ acts on the features in κ w, and the other features interact with the words only via the output of φ. Notice that this implies we can precompute and store the latent factors of the target task seen during training, so that predictions during inference can be sped-up. For example if we have two feature groups (e.g, a label and an item), first we compute the feature extraction function to the unseen items and their embeddings, and then we simply apply a dot product over the stored vectors of the labels. FIG0 shows an approach of using neural networks within factorization machines that has been proposed multiple times BID5 BID7. It replaces the dot product of factors with a learned neural function, which has been shown to improve predictive accuracy for various tasks. In this case, fast inference for cold-start documents using pre-computed label embeddings is no longer possible. It needs to store the entire neural function that takes the embeddings as inputs. Another shortcoming of replacing the dot product with a neural function is that it would no longer be possible to interpret the embeddings as containing latent factors related to the target task; There may be highly complex mappings from the embeddings to the final output via this neural function. However, it would be straightforward to combine this approach with Feat2Vec. This is not explored in this work. We can learn the the parameters of a deep factorization model θ using training data by minimizing a loss function L: DISPLAYFORM0 Here, y(x) is the true target value for x obtained from training data, andŷ(x) is the one estimated by the model; the hyperparameter γ controls the amount of regularization. For the labeling and classification tasks, we optimize the binary cross-entropy for y ∈ {0, 1}: DISPLAYFORM1 For the regression tasks where the target value is continuous, we optimize the mean squared error (MSE): DISPLAYFORM2 Neural models are typically learned using mini-batch updates, where the incremental descent is performed with respect to several instances at a time. For the implementation of this paper, we built our models using the Keras programming toolkit BID3, that is now part of Tensorflow BID0. It enables automatic differentiation, and is bundled with a generalpurpose optimization algorithm called ADAM BID12 ) that needs no tuning of gradient step sizes. It is straightforward to optimize Equation 5 directly for multiclass or binary classification. However, when the number of labels is very large, it is common practice use a binary classifier and sample the negative examples BID4. For the multi-label classification tasks, we use Feat2Vec with a binary output. In this case we would have at least two feature groups-one of the feature groups is the label that we want to predict, and the other group(s) is the input from which we want to make the prediction. The output indicates whether the label is associated with the input (y = +1), or not (y = 0). The datasets we use for our labeling experiments only contains positive labels, thus for each training example we sample a set of negative labels equal to the number of positive labels. It is typical to use one of the following sampling strategies according to the best validation error, in each case excluding the actual positive labels for each training example -(i) uniformly from all possible labels, or (ii) from the empirical distributions of positive labels. Other sampling strategies have been proposed BID24 BID23. We now discuss how Feat2Vec can be used to learn embeddings in an unsupervised setting with no explicit target for prediction. The training dataset for a Feat2Vec model consists of only the observed data. In natural language, these would be documents written by humans. Since Feat2Vec (Equation 3) requires positive and negative examples, we also need to supply unobserved data as negative examples. Consider a feature group κ i, that exists in very high dimensional space. For example, this could happen because we are modeling with one-hot encoding a categorical variable with large number of possible values. In such scenario, it is overwhelmingly costly to feed the model all negative labels, particularly if the model is fairly sparse. A shortcut around this is a concept known as implicit sampling, where instead of using all of the possible negative labels, one simply samples a fixed number (k) from the set of possible negative labels for each positively labelled record. Word2Vec makes use of an algorithm called Negative Sampling, that has little theoretical guarantees BID4. In short, their approach samples a negative observation from a noise distribution Q w2v, that is proportional to the empirical frequency of a word in the training data. We introduce a new implicit sampling method that enables learning unsupervised embeddings for structured feature sets. We can learn the correlation of features within a dataset by imputing negative labels, simply by generating unobserved records as our negative samples. Unlike Word2Vec, we do not constraint features types to be words. Features groups can be individual columns in a data matrix, but they need not to be. By grouping subfeatures using the parameter κ in Equation 3, the model can reason on more abstract entities in the data. By entity, we mean a particular feature group value. For example, in our experiments on a movie dataset, we use a "genre" feature group, where we group non-mutually exclusive indicators for movie genres including comedy, action, and drama films. We start with a dataset S + of records with | κ| feature groups. We then mark all observed records in the training set as positive examples. For each positive record, we generate k negative labels using the following 2-step algorithm:Algorithm 1 Implicit sampling algorithm for unsupervised Feat2Vec: DISPLAYFORM0 for j ∈ {1, . . ., k} do 6:x − ← x + set initially to be equal to the positive sample 7:Draw a random feature group valuex ∼ Q 2 (Xκ i, α2) 8:x − κ i ←x substitute the i-th feature type with the sampled one 9: DISPLAYFORM1 end for 11:end for 12:return S − Explained in words, our negative sampling method for unsupervised learning iterates over all of the observations of the training dataset. For each observation x +, it randomly selects the i-th feature group from a noise distribution Q 1 (·). Then, it creates a negative observation that is identical to x +, except that its i-th feature group is replaced by a value sampled from a noise distribution Q 2 (·). In our application, we use the same class of noise distributions (flattened multinomial) for both levels of sampling, but this need not necessarily be the case. We now describe the two noise distributions that we use. We use P Q (x) to denote the probability of x under a distribution Q.Sampling Feature Groups. The function params calculates the complexity of a feature extraction function φ i. To sample a feature group, we choose a feature group κ i from a multinomial distribution with probabilities proportional a feature's complexity. By complexity, we mean the number of parameters we need to learn that are associated with a particular feature group. This choice places more weight on features that have more parameters and thus are going to require more training iterations to properly learn. The sampling probabilities of each feature group are: DISPLAYFORM0 For categorical variables using a linear fully-connected layer, the complexity is simply proportional to the number of categories in the feature group. However, if we have multiple intermediate layers for some feature extraction functions (e.g., convolutional layers), these parameters should also be counted towards a feature group's complexity. The hyper-parameter α 1 helps flatten the distribution. When α 1 = 0, the feature groups are sampled uniformly, and when α 1 = 1, they are sampled proportional to their complexity. Figure A.1 in the Appendix provides a visualization of how the feature sampling rate varies with the hyperparameter for features with differing levels of complexity. Sampling Feature Group Values. To sample a value from within a feature groups κ i, we use a similar strategy to Word2Vec and use the empirical distribution of values: DISPLAYFORM1 Here, count(x) is the number of times a feature group value x appeared in the training dataset S +, and α 2 is again a flattening hyperparameter. This method will sometimes by chance generate negatively labeled samples that do exist in our sample of observed records. The literature offers two possibilities: in the Negative Sampling that Word2Vec follows, the duplicate negative samples are simply ignored BID4. Alternatively, it is possible to account for the probability of random negative labels that are identical to positively labeled data using Noise Contrastive Estimation (NCE) BID8. For our unsupervised learning of embeddings, we optimize a NCE loss function, to adjust the structural statistical modelŷ = p(y = 1| x, φ, θ), expressed in Equation 3 to account for the possibility of random negative labels that appear identical to positively labeled data. θ here represents the parameters learned in during training (i.e. the b i terms and parameters associated with the extraction functions φ i in Equation 3). Since we only deal with a dichotomous label, indicating a positive or negative sample, for unsupervised learning, we restrict our attention to usage of Equation 3 with ω as a logistic link function. An additional burden of NCE is that we need to calculate a partition function Z x for each unique record type x in the data that transforms the probabilityŷ of a positive or negative label into a wellbehaved distribution that integrates to 1. Normally, this would introduce an astronomical amount of computation and greatly increase the complexity of the model. As a work-around, we appeal to the work of BID18, who showed that in the context of language models that setting the Z x = 1 in advance effectively does not change the performance of the model. The intuition is that if the underlying model has enough free parameters that it will effectively learn the probabilities itself. Thus, it does not over/under predict the probabilities on average (since that will in penalties on the loss function).Written explicitly, the new structural probability model is: DISPLAYFORM0 where s denotes the score of a record x given parameter values/extraction functions: DISPLAYFORM1 and P Q denotes the total probability of a record x i being drawn from our negative sampling algorithm, conditional on the positively labeled record x + the negative sample is drawn for: DISPLAYFORM2 Our loss function L optimizes θ, the parameters of the feature extraction functions φ, while accounting for the probability of negative samples. DISPLAYFORM3 Feat2Vec has interesting theoretical properties. For example, it is well known that Factorization Machines can be used as a multi-label classifier: with at least two features, one can use one of the feature as the target label, and the other as the input feature to make a prediction. In such setting, the output indicates whether the label is associated with the input (y = +1), or not (y = 0), and therefore the input can be associated with more than one label. With n feature types, Feat2Vec is equivalent to optimizing a convex combination of the loss functions from n individual Factorization Machines. In other words, it optimizes n multi-label classifiers, where each classifier is optimized for a different target (i.e.,a specific feature group). We show the proof of this in the Appendix 1. We now address our working hypotheses for evaluating supervised embeddings. For all our experiments we define a development set and a single test set which is 10% of the dataset, and a part of the development set is used for early stopping or validating hyper-parameters. Since these datasets are large and require significant time to train on an Nvidia K80 GPU cluster, we report on only a single training-test split. For the multi-label classification task in 4.1.1 we predict a probability for each document-label pair and use an evaluation metric called Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC). Since we only observe positive labels, for each positive label in the test set we sample negative labels according to the label frequency. This ensures that if a model merely predicts the labels according to their popularity, it would have an AUC of 0.5. A caveat of our evaluation strategy is that we could be underestimating the performance of our models-there is a small probability that the sampled negatives labels are false negatives. However, since we apply the same evaluation strategy consistently across our methods and baselines, the relative difference of the AUC is meaningful. We choose the AUC as a metric because it is popular for both classification and ranking problems. For the regression task in 4.1.2, we use mean squared error (MSE) as the evaluation metric. In preliminary experiments we noticed that regularization slows down convergence with no gains in prediction accuracy, so we avoid overfitting only by using early stopping. We share most of the code for the experiments online 1 for reproducibility. For our feature extraction function φ for text, we use a Convolutional Neural Network (CNN) that has been shown to be effective for natural language tasks BID11 BID28. In Appendix B we describe this network and its hyper-parameters. Instead of tuning the hyper-parameters, we follow previously published guidelines BID30. We compare Feat2Vec with an extension of matrix factorization that can generalize to unseen items for text documents, Collaborative Topic Regression , a method with an open-source Python implementation 2. We evaluate them on the CiteULike dataset which consists of pairs of scientific articles and the users who have added them to their personal libraries, and it contains 16,980 unique articles and 5,551 unique users. We use the models to predict users who For CTR we use the hyper-parameters reported by the authors as best, except for r which we found had a significant impact on training time. We only consider r ∈ {5, 10, 15} and choose the value which gives the best performance for CTR (details in Appendix A.2). On the warm-start condition, CTR has an AUC of 0.9356; however, it shows significant degradation in performance for unseen documents and it only performs slightly better than random chance with an AUC of 0.5047. On the other hand, Feat2Vec achieves AUC of 0.9401 on the warm-start condition, and it only degrades to 0.9124 on unseen documents. Feat2Vec can also be trained over ten times faster, since it can leverage GPUs. 4 We also note that we have not tuned the architecture or hyper-parameters of the feature extraction function φ and greater improvements are possible by optimizing them. We now compare with a method called DeepCoNN, a deep network specifically designed for incorporating text into matrix factorization BID31 -which reportedly, is the state of the art for predicting customer ratings when textual reviews are available. For Feat2Vec we use the same feature extraction function (see Appendix B.1 for details) used by DeepCoNN. We evaluate on the Yelp dataset 5, which consists of 4.7 million reviews of restaurants. For each user-item pair, DeepCoNN concatenates the text from all reviews for that item and all reviews by that user. The concatenated text is fed into a feature extraction function followed by a factorization machine. In contrast, for Feat2Vec, we build 3 feature groups: item identifiers (in this case, restaurants), users and review text. TAB0 compares our methods to DeepCoNN's published because a public implementation is not available. We see that Feat2Vec provides a large performance increase when comparing the reported improvement, over Matrix Factorization, of the mean squared error. Our approach is more general, and we claim that it is also more efficient. Since DeepCoNN concatenates text, when the average reviews per user isn u and reviews per item isn i, each text is duplicated on averagen i ×n u times per training epoch. In contrast, for Feat2Vec each review is seen only once per epoch. Thus it can be 1-2 orders of magnitude more efficient for datasets wheren i ×n u is large. Ex ante, it is unclear to us how to evaluate the performance of an unsupervised embedding algorithm, but we felt that a reasonable task would be a ranking task one might practically attempt using our datasets. This task will assess the similarity of trained embeddings using unseen records in a left-out dataset. In order to test the relative performance of our learned embeddings, we train our unsupervised Feat2Vec algorithm and compare its performance in a targeted ranking task to Word2Vec's CBOW algorithm for learning embeddings. In our evaluation approach, we compare the cosine similarity of the embeddings of two entities where these entities are known to be associated with each other since they appear in the same observation in a test dataset. In particular, in the movie dataset 3 For warm-start we use https://www.cs.cmu.edu/˜chongw/data/citeulike/folds/ cf-train-1-items.dat and for cold-start predictions, we use the file ofm-train-1-items.dat and the corresponding test sets for each 4 Feat2Vec and MF were trained on an Nvidia K80 GPU, while CTR was trained on a Xeon E5-2666 v3 CPU.5 https://www.yelp.com/dataset/challenge we compare the similarity of movie directors to those of actors who were cast in the same film for a left-out set of films. For our educational dataset, we compare rankings of textbooks by evaluating the similarity of textbook and user embeddings. We evaluate the rankings according to their mean percentile rank (MPR): DISPLAYFORM0 where R i is the rank of the entity under our evaluation procedure for observation i. This measures on average how well we rank actual entities. A score of 0 would indicate perfect performance (i.e. top rank every test sample given), so a lower value is better under this metric. See the appendix §A.1 for further details on the experimental setup. Movies The Internet Movie Database (IMDB) is a publicly available dataset 6 of information related to films, television programs and video games. Though in this paper, we focus only on data on its 465,136 movies. TAB2.1 in the appendix (§A.1) summarizes the feature types we use. It contains information on writers, directors, and principal cast members attached to each film, along with metadata. Education We use a dataset from an anonymized leading technology company that provides educational services. In this proprietary dataset, we have 57 million observations and 9 categorical feature types which include textbook identifier, user identifier, school identifier, and course the book is typically used with, along with other proprietary features. Here, each observation is an "interaction" a user had with a textbook. After training, we use the cast members associated with the movies of the test set and attempt to predict the actual director the film was directed. We take the sum of the cast member embeddings, and rank the directors by cosine similarity of their embeddings to the summed cast member vector. If there is a cast member in the test dataset who did not appear in the training data, we exclude them from the summation. For the educational dataset, we simply use the user embedding directly to get the most similar textbooks. TAB1 presents the from our evaluation. Feat2Vec sizably outperforms CBOW in the MPR metric. In fact, Feat2Vec predicts the actual director 2.43% of the times, while CBOW only does so 1.26% of the time, making our approach almost 2 times better in terms of Top-1 Precision metric. We explore in greater detail the distribution of the rankings in the appendix in §A.2. We now focus on how well Feat2Vec performs on a real-valued feature with a complex feature extraction function. We expect this task to highlight Feat2Vec's advantage over token-based embedding learning algorithms, such as Word2Vec, since our rating embedding extraction function will require embeddings of numerically similar ratings to be close, while Word2Vec will treat two differing ratings tokens as completely different entities. We evaluate the prediction of the real-valued rating of movies in the test dataset by choosing the IMDB rating embedding most similar 7 to the embedding of the movie's director, and compute the Root Mean Squared Error (RMSE) of the predicted rating in the test dataset. We also vary α 1, the flattening hyperparameter for feature group 6 http://www.imdb.com/interfaces/ 7 As before, the metric is cosine similarity. sampling, to see what effect this hyperparameter has on our performance. Intuitively, a low α 1 will greatly improve the quality of the ratings embeddings learned, since it has relatively few parameters and is otherwise sampled infrequently. At the same time, with low α 1 the director feature will be sampled less since it is one of the most complex features to learn, so the learned director embeddings may be of poorer quality. FIG2 displays the of our experiment, benchmarked against the performance of Word2Vec's CBOW algorithm in the prediction task. We also show as a baseline the RMSE of a random uniform variable over the range of possible ratings (0 to 10). As is evident from the plot, CBOW performs a bit better than a random prediction, but is also handily outperformed by Feat2Vec across all hyper-parameter settings. The algorithm's performance does not seem very sensitive to the hyperparameter choice. The original Factorization Machine formulation has been extended for multiple contexts. For example, Field-Aware Factrorization Machine BID10 allows different weights for some feature interactions, but does not allow feature groups or feature extraction functions like Feat2Vec does. Algorithms that calculate continuous representations of entities other than words have been proposed for biological sequences BID1, of vertices in network graphs BID19 or in machine translation for embeddings of complete sentences BID13. Generative Adversarial Networks BID6 )(GANs) have been used to produce unsupervised embeddings of images effective for classification BID21 and for generating natural language BID20. To our knowledge, GANs have not been used for jointly embedding multiple feature types. Adversarial training could be an alternative to NCE for unsupervised learning, but we leave this for future study. We recently discovered a promising direction for an algorithm still in development called StarSpace BID29 with similar goals from ours. Even though they intend to be able to embed all types of features, at the time of the writing of this paper, their pre-print method was limited to only work for bag of words. While Feat2Vec can jointly learn embeddings for all feature values in a dataset, StarSpace samples a single arbitrary feature. Our preliminary experiments suggest that sampling a single feature does not produce embeddings that generalize well. Nonetheless, a limitation of our work is that we do not compare with StarSpace, which future work may decide to do. Embeddings have proven useful in a wide variety of contexts, but they are typically built from datasets with a single feature type as in the case of Word2Vec, or tuned for a single prediction task as in the case of Factorization Machine. We believe Feat2Vec is an important step towards generalpurpose methods, because it decouples feature extraction from prediction for datasets with multiple feature types, it is general-purpose, and its embeddings are easily interpretable. In the supervised setting, Feat2Vec is able to calculate embeddings for whole passages of texts, and we show experimental outperforming an algorithm specifically designed for text-even when using the same feature extraction CNN. This suggests that the need for ad-hoc networks should be situated in relationship to the improvements over a general-purpose method. In the unsupervised setting, Feat2Vec's embeddings are able to capture relationships across features that can be twice as better as Word2Vec's CBOW algorithm on some evaluation metrics. Feat2Vec exploits the structure of a datasets to learn embeddings in a way that is structurally more sensible than existing methods. The sampling method, and loss function that we use have interesting theoretical properties. To the extent of our knowledge, Unsupervised Feat2Vec is the first method able to calculate continuous representations of data with arbitrary feature types. Future work could study how to reduce the amount of human knowledge our approach requires; for example by automatically grouping features into entities, or by automatically choosing a feature extraction function. These ideas can extend to our codebase that we make available 8. Overall, we evaluate supervised and unsupervised Feat2Vec on 2 datasets each. Though further experimentation is necessary, we believe that our are an encouraging step towards general-purpose embedding models. Bag of categories 244,241 "George Johnson", "Jack Russell" Principal cast members (actors) Bag of categories 1,104,280 "George Clooney", "Brad Pitt", "Julia Roberts" A APPENDIXES A.1 UNSUPERVISED RANKING EXPERIMENT DETAILS For our evaluation, we define a testing set that was not used to tune the parameters of the model. For the IMDB dataset, we randomly select a 10% sample of the observations that contain a director that appears at least twice in the database 9. We do this to guarantee that the set of directors in the left-out dataset appear during training at least once, so that each respective algorithm can learn something about the characteristics of these directors. For the educational dataset, our testing set only has observations of textbooks and users that appear at least 10 times in training. For both Feat2Vec and CBOW, we perform cross-validation on the loss function, by splitting the 10% of the training data randomly into a validation set, to determine the number of epochs to train, and then train the full training dataset with this number of epochs.10 While regularization of the embeddings during training is possible, this did not dramatically change , so we ignore this dimension of hyperparameters. We rank left-out entity pairs in the test dataset using the ordinal ranking of the cosine similarity of target and input embeddings. For the IMDB dataset, the target is the director embedding, and the input embedding is the sum of the cast member embeddings. For the educational dataset, the target is the textbook embedding, and the input embedding is the user embedding. For training Feat2Vec we set α 1 = α 2 = 3/4 in the IMDB dataset; and α 1 = 0 and α 2 = 0.5 for the educational. In each setting, α 2 is set to the same flattening hyperparameter we use for CBOW to negatively sample words in a document. We learn r = 50 dimensional embeddings under both algorithms. Below we describe how CBOW is implemented on our datasets for unsupervised experiments and what extraction functions are used to represent features in the IMDB dataset. Word2Vec For every observation in each of the datasets, we create a document that tokenizes the same information that we feed into Feat2Vec. We prepend each feature value by its feature name, and we remove spaces from within features. In Figure A.2 we show an example document. Some features may allow multiple values (e.g., multiple writers, directors). To feed these features into the models, for convenience, we constraint the number of values, by truncating each feature to no more than 10 levels (and sometimes less if reasonable). This in retaining the full set of information for well over 95% of the values. We pad the sequences with a "null" category whenever necessary to maintain a fixed length. We do this consistently for both Word2Vec and Feat2Vec. We use the CBOW Word2Vec algorithm and set the context window to encompass all other tokens in a document during training, since the text in this application is unordered. Here, we explain how we build these functions:• Bag of categories, categorical, and boolean: For all of the categorical variables, we learn a unique r-dimensional embedding for each entity using a linear fully-connected layer (Equation 4). We do not require one-hot encodings, and thus we allow multiple categories to be active; ing in a single embedding for the group that is the sum of the embeddings of the subfeatures. This is ordering-invariant: the embedding of "Brad Pitt" would be the same when he appears in a movie as a principal cast member, regardless whether he was 1st or 2nd star. Though, if he were listed as a director it may in a different embedding.• Text: We preprocess the text by removing non alpha-numeric characters, stopwords, and stemming the remaining words. We then follow the same approach that we did for categorical variables, summing learned word embeddings to a "title embedding" before interacting. It would be easy to use more sophisticated methods (e.g, convolutions), but we felt this would not extract further information.• Real-valued: For all real-valued features, we pass these features through a 3-layer feedforward fully connected neural network that outputs a vector of dimension r, which we treat as the feature's embedding. Each intermediate layer has r units with relu activation functions. These real-valued features highlight one of the advantages of the Feat2Vec algorithm: using a numeric value as an input, Feat2Vec can learn a highly nonlinear relation mapping a real number to our high-dimensional embedding space. In contrast, Word2Vec would be unable to know ex ante that an IMDB rating of 5.5 is similar to 5.6. Figure A.3 shows the full distribution of rankings of the IMDB dataset, rather than summary statistics, in the form of a Cumulative Distribution Function (CDF) of all rankings calculated in the test dataset. The graphic makes it apparent for the vast majority of the ranking space, the rank CDF of Feat2Vec is to the left of CBOW, indicating a greater probability of a lower ranking under Feat2Vec. This is not, however, the case at the upper tail of ranking space, where it appears CBOW is superior. However, when we zoom-in on the absolute upper region of rankings (1 to 25), which might be a sensible length of ranks one might give as actual recommendatiosn, it is the case that up until rank 8 or so, Feat2Vec outperforms CBOW still. Intermediate rankings are still strong signals that our Feat2Vec algorithm is doing a better job of extracting information into embeddings, particularly those entities that appear sparsely in the training data and so are especially difficult to learn. Theorem 1. The gradient for learning embeddings with Feat2Vec is a convex combination of the gradient from n targeted Factorization Machines for each feature in the data when each feature group is a singleton, where n is the total number of features in the dataset. Proof. Let S + κi denote the positively labeled records whose corresponding negative samples resample feature κ i. For convenience, suppress the inclusion of learned parameters θ in the notation in this section while understanding the feature extraction functions φ implicitly include these parameters. We can express the loss function L, the binary cross-entropy of the data given the Feat2Vec model, as follows: DISPLAYFORM0 is simply the probability of the record's feature value x f under the second step noise distribution Q 2 (X f, α 2): DISPLAYFORM1 We now drop the term containing the probability of assignment to a feature group p(x + ∈ S + κi) since it is outside of the learned model parameters φ and fixed in advance: DISPLAYFORM2 Thus, the loss function is just a convex combination of the loss functions of the targeted classifiers for each of the p features, and by extension so is the gradient since: DISPLAYFORM3 Thus the algorithm will, at each step, learn a convex combination of the gradient for a targeted classifier on feature f, with weights proportional to the feature group sampling probabilities in step 1 of the sampling algorithm. Note that if feature groups are not singletons, the gradient from unsupervised Feat2Vec will analogously be a convex combination of n gradients learned from supervised learning tasks on each of the n feature groups. Figure A.4: Feature extraction network used for labelling tasks. We use f=1000 convolutional filters each of width 3 (words) Here we describe the details of the feature extraction function φ used in our experiments for supervised tasks in §4.1. An overview of the network is given in Fig. A.4. We choose the most common words of each dataset to build a vocabulary of size n, and convert the words of each document to a sequence of length t of one-hot encodings of the input words. If the input text is shorter than t, then we pad it with zeros; if the text is longer, we truncate it by discarding the trailing words. Therefore, for a vocabulary size n, the input has dimensions t × n. These t× dimensional matrix is then passed through the following layers:1. We use an embedding layer to assign a d-dimensional vector to each word in the input passage of text. This is done through a d × n-dimensional lookup table, which in an t × d matrix.2. We extract features from the embeddings with functions called convolutional filters BID16 (also called feature maps). A convolutional filter is simply a matrix learned from an input. We learn f filters that are applied on groups of m adjacent word embeddings, thus each of our filters is a d × m matrix of learned parameters. Filters are applied by computing the element-wise dot product of the filter along a sliding window of the entire input. The ing output for each filter is a vector of length t − m + 1. We also apply a ReLU activation to the output of each filter.3. Consider the case of inputs of different lengths. For very short texts, the output of the filters will be mostly zero since the input is zero-padded. To enforce learning from the features of the text, and not just its length we apply a function called 1-max pooling to the output of the filters: from the t − m + 1 output vector of each filter, we select the maximum value. This yields a vector of length F, a representation of the passage which is independent of its length.4. We learn higher-level features from the convolutional filters. For this, we use a fully connected layer with p units and a ReLU activation, 5. During training (not in inference), we prevent the units from co-adapting too much with a dropout layer BID26. Dropout is a form of regularization that for each mini-batch randomly drops a specified percentage of units.6. the final embedding for x j (that is used in the factorization) is computed by a dense layer with r output units and an activation function, where r is the embedding size of our indexable items. We set the maximum vocabulary size n to 100,000 words, and input embedding size d to 50 for all experiments. We initialize the input word embeddings and the label embeddings using Word2Vec BID17 We have have not evaluated multiple architectures or hyperparameter settings and obtain good on diverse datasets with the same architecture, which was designed followed recommendations from a large scale evaluation of CNN hyper parameters BID30. We set the number of convolutional filters f to 1,000, and the dropout rate to 0.1. The maximum sequence length t was chosen according to the typical document length (350 words for CiteULike and 250 for Yelp). For the CTR dataset, because we use very small values of r, due to the tendency of the ReLU units to'die' during training (output zero for all examples), which can have a significant impact, we used instead PReLU activations BID9 for the final layer, since they do not suffer from this issue. The CNN architecture used for DeepCoNN BID31 ) is similar to the previous section. It consists of a word embedding lookup table, convolutional layer, 1-max pooling and a fully connected layer. We use the hyper-parameters that the authors report as best -100 convolution filters and 50 units for the fully connected layer. We set the word embedding size to 100, the vocabulary size to 100,000 and the maximum document length to 250. To compare Feat2Vec with Collaborative Topic Regression, we choose the embedding size r ∈ {5, 10, 15} for which CTR performs best. The are show in TAB2.2.

Learn dense vector representations of arbitrary types of features in labeled and unlabeled datasets

We use a hypernetwork to predict optimal weights given hyperparameters, and jointly train everything together.