Modeling Embedding Dimension Correlations via Convolutional Neural Collaborative Filtering
Xiaoyu Du Xiangnan He Fajie Yuan Jinhui Tang Zhiguang Qin Tat-Seng Chua




Convolutional NCF]Modeling Embedding Dimension Correlations via Convolutional Neural Collaborative Filtering






duxy.me@gmail.com

  University of Electronic Science and Technology of China
  Chengdu
  Sichuan
  China



Xiangnan He is the corresponding author.
xiangnanhe@gmail.com

  University of Science and Technology of China
  Hefei
  Anhui
  China



fajieyuan@tencent.com

  Platform and Content Group (PCG) of Tencent
  Shenzhen
  Guangdong
  China



jinhuitang@njust.edu.cn

  Nanjing University of Science and Technology
  Nanjing
  Jiangsu
  China



qinzg@uestc.edu.cn

  University of Electronic Science and Technology of China
  Chengdu
  Sichuan
  China


dcscts@nus.edu.sg

  National University of Singapore
  Singapore
  Singapore










	As the core of recommender system, collaborative filtering (CF) models the affinity between a user and an item from historical user-item interactions, such as clicks, purchases, and so on. Benefited from the strong representation power, neural networks have recently revolutionized the recommendation research, setting up a new standard for CF. However, existing neural recommender models do not explicitly consider the correlations among embedding dimensions, making them less effective in modeling the interaction function between users and items. In this work, we emphasize on modeling the correlations among embedding dimensions in neural networks to pursue higher effectiveness for CF. We propose a novel and general neural collaborative filtering framework, namely ConvNCF, which is featured with two designs: 1) applying outer product on user embedding and item embedding to explicitly model the pairwise correlations between embedding dimensions, and 2) employing convolutional neural network above the outer product to learn the high-order correlations among embedding dimensions. To justify our proposal, we present three instantiations of ConvNCF by using different inputs to represent a user and conduct experiments on two real-world datasets. Extensive results verify the utility of modeling embedding dimension correlations with ConvNCF, which outperforms several competitive CF methods.
	
	
	
	
	







<ccs2012>
<concept>
<concept_id>10002951.10003260.10003261.10003269</concept_id>
<concept_desc>Information systems Collaborative filtering</concept_desc>
<concept_significance>500</concept_significance>
</concept>
<concept>
<concept_id>10002951.10003317.10003347.10003350</concept_id>
<concept_desc>Information systems Recommender systems</concept_desc>
<concept_significance>500</concept_significance>
</concept>
<concept>
<concept_id>10010147.10010257.10010293.10010294</concept_id>
<concept_desc>Computing methodologies Neural networks</concept_desc>
<concept_significance>500</concept_significance>
</concept>
</ccs2012>


[500]Information systems Collaborative filtering
[500]Information systems Recommender systems
[500]Computing methodologies Neural networks



















[
    Tat-Seng Chua
    December 30, 2023
=====================





§ INTRODUCTION



Recommender system plays a vital role in the era of information explosion. It not only alleviates the information overload issue and facilities the information seeking for users, but also serves as an effective solution to increase the traffic and revenue for service providers. Given its extensive use in Web applications like E-commerce <cit.>, social media <cit.>, news portal <cit.>, and music sites <cit.>, its importance cannot be overstated as a highly valuable learning system. For example, it is reported that about 30% traffic in Amazon are brought by recommendations <cit.>, and the Netflix recommender system has contributed over $1 billion revenue per year <cit.>.


Modern recommender systems typically rely on collaborative filtering (and/or its content/context -aware variants) to predict a user's preference on unknown items <cit.>. The learning objective can be abstracted as estimating the affinity score between a user and an item, such that the recommendation list for a user can be obtained by ranking the candidate items according to the affinity scores. Generally speaking, the key to build a CF model lies in twofold: 1) how to represent a user and an item, and 2) how to build the predictive function based on user and item representations. As a pioneering CF model, matrix factorization (MF) <cit.> embeds a user and an item into vector representations, and formulates the predictive function as the inner product between user embedding vector item embedding vector.




Soon after MF was introduced to recommender system, it becomes prevalent in recommendation research with many variants developed. Some representative  variants include FISM (factored item similarity model) <cit.> and SVD++ <cit.> which enrich the user embedding with the embeddings of the user's interacted items, and FM (factorization machine) <cit.> and SVDfeature <cit.> which extend the predictive function with the inner product between the embeddings of content (and/or context) features. 


Without exaggeration, we would say that factorization methods are the most popular approach and play a dominant role in recommendation research in the past decade. In essence, MF leads to the de facto standard for modeling the CF effect — measuring the user-item affinity with inner product in the embedding space. While inner product is effective in capturing the low-rank structure in sparse user-item interaction data <cit.>, its simplicity (i.e., no learnable parameters are involved) and linearity (i.e., no any nonlinear transformations) limit the expressive power of the predictive function. For example, He et al. <cit.> demonstrated that it may lead to unexpected ranking loss when the embedding size is not sufficiently large, and Hsieh et al. <cit.> illustrated the cases that inner product may fail due to its violation of the triangle inequality. 




To address the limitations of factorization methods and move towards the next generation of recommender systems, neural network models have been explored for CF in recent years. To date, most neural recommender models can be described within the neural collaborative filtering (NCF) framework <cit.>, which use neural networks to emphasize on either the user/item representation learning part <cit.> or the predictive function part <cit.>. Although these methods have achieved substantial improvements, we argue a common drawback is that, they forgo considering the correlations among embedding dimensions in the predictive function. To be specific, the mainstream design of the predictive function is to place a multi-layer perceptron (MLP) above the concatenation (which retains the original information) <cit.> or element-wise product (which subsumes the inner product) <cit.> of user embedding and item embedding. Apparently, both operations assume the embedding dimensions are independent of each other, that is, no correlations among the dimensions are captured. Arguably, the following MLP network can approximate any continuous function <cit.>, therefore it might be capable of capturing the possible correlations. However, such process is rather implicit and as such, it may be ineffective in capturing certain relations — an evidence is from <cit.> showing that much more parameters have to be used in order to approximate the simple multiplicative relation.












In this work, we highlight the importance of modeling the correlations among embedding dimensions for CF, proposing a new CF solution named Convolutional Neural Collaborative Filtering (ConvNCF). 
As illustrated in Figure <ref>,
ConvNCF has two characteristics making it distinct from existing models: 1) above user embedding and item embedding, we employ outer product (rather than concatenation or inner product) so as to explicitly capture the pairwise correlations between embedding dimensions, and 2) above the matrix generated by outer product, we employ convolution neural network (CNN) so as to learn high-order correlations in a hierarchical way. As ConvNCF concerns only the design of the predictive function, it is a general framework that can be specified with any embedding function that results in user embedding and item embedding. To show this universality, we devise three instantiations of ConvNCF by using the embedding function of three classical CF models — 1) MF <cit.>, which projects a user's ID into user embedding, 2) FISM <cit.>, which projects a user's interacted items into user embedding, and 3) SVD++ <cit.>, which projects a user's ID and interacted items into user embedding. We term the three specific methods as ConvNCF-MF, ConvNCF-FISM, and ConvNCF-SVD++, respectively, and conduct experiments on two real-world datasets to explore their effectiveness. Comparative results show that our ConvNCF methods outperform state-of-the-art CF methods in item recommendation, and extensive ablation studies verify the usefulness of both outer product and CNN in modeling embedding dimension correlations. To facilitate the research community, we have released the codes in: https://github.com/duxy-me/ConvNCFhttps://github.com/duxy-me/ConvNCF.



Note that a preliminary version of this work has been published as a conference paper in IJCAI 2018 <cit.>. We summarize the main changes as follows:


  * Introduction (Section <ref>). We reconstruct the abstract and introduction to emphasize the motivation of this extended version.


  * Methods (Section <ref> and Section <ref>). We present ConvNCF as a general CF framework, and specify three methods that differ in the user embedding function to demonstrate its universality. The preliminary version only presents one specific method. 


  * Experiments (Section <ref>). This section is complemented with results of the two additional methods — ConvNCF-FISM and ConvNCF-SVD++ — to further justify the effectiveness of our proposal. 


  * Preliminaries (Section <ref>) and Related Work (Section <ref>). The two sections are newly added to make the paper more complete and self-contained. 













The main contributions of this paper are as follows. 

	
  * We propose a neural network framework named ConvNCF for CF, which explicitly models embedding dimension correlations and uses CNN to learn high-order correlations from locally to globally in a hierarchical way. 
	
  * We implement three instantiations of ConvNCF that use different embedding functions to demonstrate the universality and effectiveness of ConvNCF. 
	
	
  * This is the first work that explores the utility of capturing the correlations among embedding dimensions, providing a new path to improve recommendation models. 






§ RELATED WORK


This work lies in the topic of neural collaborative filtering. In this section, we first give a review of collaborative filtering on the embedding and interaction function, and then introduce the neural network based collaborative filtering. At last, to deepen the comprehension over neural network, we demonstrate the latest CNN developments and its applications in recommender systems.



 §.§ Collaborative Filtering

User behaviors on the online platforms (purchasing, browsing or commenting) imply user preferences. By building the user-item interactions through these behavior records, Collaborative Filtering (CF) is able to mine user hidden preferences. According to the types of user behaviors, CF can  be usually classified into two categories, namely explicit and implicit feedback based CF. Explicit feedback data such as user ratings directly indicates user's active evaluation. It has been a significant research task in the last decades.
To estimate the accurate score toward a specific item, various factorization models with a regression loss have been proposed. Particularly, models such as SVD++ <cit.>, Localized MF <cit.>, Hierarchical MF <cit.>, Social-aware MF <cit.>, and CrossPlatform MF <cit.> have gained great success on specific tasks due to the ability to model specific contextual features. 


 Another type of interactions is known as implicit feedback, which records any user behaviors (e.g., what they watch and what item they buy). Implicit feedback data is more prevalent in practice since
 it does not require the user to express his taste explicitly <cit.>. Hence, in this paper we aim to build recommendation algorithms based on implicit user feedback.





In most practical recommender systems, users usually only focus on the top ranked items rather than all rating scores. 

From this perspective, CF with implicit feedback seems like a personalized ranking problem rather than a score predicting problem. To address this problem, BPR <cit.> loss is proposed to model the relative preferences between a pair of interactions, one of which is observed while the other is unobserved. The predicting score of the observed interaction must be higher than that of the unobserved one. Through this ranking scheme, BPR successfully trains a number of CF models <cit.>,

including both shallow factorization models <cit.> and deep neural network model <cit.>.

In fact, BPR is currently the dominant loss for CF models and has many improvements, such as improvedBPR <cit.> by changing  the negative sampling to select informative examples and APR <cit.> by applying adversarial learning to enhance the model robustness.



 §.§ Matrix Factorization

Matrix Factorization (MF) is a significant technique in many domains <cit.> due to its ability to distill co-occurrence patterns <cit.>. 
Within the variant CF models, MF is also an important class. Traditional MF algorithms work by decomposing the user-item interaction matrix into inner product of two lower-rank matrices <cit.>. Recently, the models representing users and items with two lower-rank matrices and predicting the interactions by inner product are considered to be MF family. Its subsequent works mainly focus on devising the user embedding and item embedding. 

The earliest MF model is FUNK-SVD[http://sifter.org/ simon/journal/20061211.html] that assigns a latent vector to each of the users and items. The prediction of FUNK-SVD is the inner product of the vectors. Despite the name, FUNK-SVD apply no singular value decomposition to get the model but use learning-based approach. Thus the embedding is easily extended with more complex structures. FISM <cit.> combines the set of features from interacted items as user embedding. NAIS <cit.> weights the addition during combining process. 
SVD++ <cit.> synthesizes latent vectors and the item correlations to construct hybrid embedding. DeepMF <cit.> applies MLP <cit.> over the original embedding to abstract a high-level embedding. Additionally, the embedding could be generated with the content and context features <cit.>. VBPR <cit.> takes latent vector composed with image features extracted by AlexNet <cit.> as item embedding. CCF <cit.> proposes a content-based CF for news topic. Music recommendation <cit.> integrates the content-based feature to express the song. Some works even incorporate external knowledge <cit.>. It is notable that some of these models were proposed for explicit feedback but they also perform well for implicit feedback by training with BPR loss <cit.>.




 §.§ Neural Collaborative Filtering

Neural network is known as a powerful data-based model well-performed in a wide range of domains. It is always described as a pipeline of layers. To meet the characteristics of different tasks, various novel structures of the neural network are proposed recently <cit.>. Multi-layer Perceptron (MLP) <cit.> is the fundamental neural network, the main body of which is composed of fully-connected layers, that outputs a projection of input features. Convolutional nerual network (CNN) <cit.> reduces the number of parameters of MLP and increases the number of layers with convolution operations, which automatically extract partial projection with convolution kernel. Attention mechanism <cit.> adapts the feature weights based on some auxiliary information, in order to capture more effective features. Due to the extraordinary performance on images, CNN has become the dominant 
module in the domain of computer vision.


More than that, both MLP and CNN models are widely used in the
 natural language processing <cit.> and recommender system <cit.> domains.

Neural collaborative filtering (NCF) is a family of models using neural network for the CF task. Most NCF methods focus on devising powerful interaction functions above the embedding layer instead of using the simple inner product. A popular substitute of inner product is to improve the element-wise product, such as weighted element-wise product <cit.>, or a MLP over the element-wise product <cit.>.

In the NCF task, MLP <cit.>  has been extensively investigated in recent years, due to its simplicity and effectiveness. The basic idea of MLP is to stack multiple fully-connected layers, each of which are usually followed by a non-linear activation layer. 
For example, NeuMF <cit.> devises a two-path model that one of the path is a MLP on top of the concatenation of user embedding and item embedding, and the other is a MLP above element-wise product of the embeddings. JRL <cit.> takes a MLP over the element-wised product of embeddings as input. Similarly, deepMF <cit.> learns a high-level embedding with MLP. CDAE <cit.> on the other way devises an auto-encoder with fully-connected layers. Despite the success of MLP, its deficiency cannot be ignored. The models with MLP are easy to overfit and usually need more computing resources due to the large amount of parameters.  




 §.§ Convolutional Neural Network

In this work, we mainly investigate the utility of convolutional neural network (CNN) in the CF task.

The core of CNN is known as a partial feature extractor (convolutional layer). In each convolution layer, the output feature is not a projection of the whole input layer but several maps composed of partial features generated by convolution kernels. The key to obtain effective features with fewer parameters is that the partial features in the same map share a group of convolution kernels. Thus, compared with the fully-connected layer used in MLP, convolutional layer has much fewer parameters, which leads to more robust learning process and takes up fewer computing resources. 



Due to these advantages, we believe CNN has the potential to substitute the MLP in the CF task with less computing cost and better recommendation accuracy.

In fact, several works that integrate CNN within the recommendation models have achieved success. The key idea of these works is to improve the original methods by feeding the recommemnder system with additional CNN features. One typical work is VBPR <cit.> that leverages the image features extracted with Alexnet <cit.> to express the items more accurately. Similarly, ConvMF <cit.> generates document latent features with a textual CNN. Though these works can provide more accurate auxiliary information, they rarely change the core pattern in predicting preference from the model perspective. 
NextItNet <cit.> is a newly proposed CNN model for the session-based recommendation. 
It combines masked filters with 1D dilated convolutions <cit.> to increase the receptive fields when modeling long-range session data. Since this work targets at building a generic recommendation framework for traditional CF scenarios, we omit the detailed discussion of NextItNet.




§ PRELIMINARIES


This section presents some technical preliminaries to the work. We first introduce the problem formulation of recommendation, and then discuss the recently proposed neural collaborative filtering framework <cit.>. For the ease of reading, we use bold uppercase letter (P) to denote a matrix, bold lowercase letter to denote a vector (p_u), and calligraphic uppercase letter to denote a tensor (ℰ).
Moreover, scalar p_u,k denotes the (u,k)-th element of matrix P, and vector p_u denotes the u-th row vector in P.
Let ℰ be 3D tensor, then scalar e_a,b,c denotes the (a,b,c)-th element of tensor ℰ, and vector e_a,b denotes the slice of ℰ at the element (a,b). Table <ref> and Table <ref> summarize the brief
parameters and abbreviations used in this paper, respectively.









 §.§ Problem Formulation



Recommendation is a large-scale personalized ranking task, which generates a distinct ranking list for each user. In model-based CF methods, the items are ranked by a predictive model ŷ_ui, which estimates how likely the user u will consume the item i. That is, ŷ_ui gives a high score if the user u is likely to consume the item i, otherwise it gives a low score.

The predictive function ŷ_ui is trained by learning parameters on the observed user-item interactions. Let M and N denote the number of users and items respectively, the observed user-item interaction data is represented as a matrix Y∈ℝ^M× N. The element y_ui located at u-th row and i-th column denotes the preference of user u on item i. In most real-world scenarios, the interaction data is implicit feedback that records user behaviours, such as clicking, browsing, purchasing, etc. In this case, y_ui is usually expressed as a binary variable to denote the user preference:

    y_ui = 
    
        1    if  (u,i)  is observed,
    
        0    otherwise.




After the model is trained to fit the interaction data well, during the testing phase that recommends items to user u, we evaluate ŷ_ui on all candidate items (e.g., items that are not interacted by the user or promotion items), and rank the items based on the values. 




 §.§ Neural Collaborative Filtering






There are two key ingredients in developing a CF model: 1) how to represent a user and an item, and 2) how to build the predictive function based on user and item representations.
In most cases, the users and items are represented as embeddings. Since we will present common embedding methods for CF in Section <ref>, here we discuss more on the predictive function.  





Neural collaborative filtering (NCF) <cit.> proposes to parameterize the predictive function with feed-forward neural networks. Owing to the strong representation ability of neural networks in theory, NCF has been recognized as an effective approach to model user-item interactions and become a prevalent choice <cit.>.

The predictive function in NCF can be abstracted as follows <ref>:

    ϕ_0    = ℱ(f^U(u),f^I(i)), 
    ϕ_l    = ℒ_l(ϕ_l-1),   for   l = 1...L 
    ŷ_ui   = w^T ϕ_L,

where f^U(·) and f^I(·) denote the embedding function for users and items, respectively; ℱ(·) denotes the merge function that combines user embedding and item embedding to feed to hidden layers; ℒ_l denotes the transformation function of the l-th hidden layer, ϕ_l denotes the output of the l-th layer; ϕ_L denotes the output of the last layer, and vector w is to project ϕ_L to the final prediction score. Note that NCF presents a general framework such that each module in the framework is subjected to design. 





Next, we concentrate on the merge function ℱ(·) used in existing methods. 

The two most prevalent choices in recommendation literature <cit.> are element-wise product and concatenation. 

Element-wise Product generates the combined vector by multiplying the corresponding dimension of user embedding and item embedding, that is, 

    ℱ(f^U(u),f^I(i)) = 
    (
    
        f^U(u)_1· f^I(i)_1,
    
        f^U(u)_2· f^I(i)_2,
    ⋯, 
    
        f^U(u)_K· f^I(i)_K
    )

The rationality of this operation for CF stems from inner product — by summing the elements in the resultant vector, it is equivalent to the output of inner product. 
Note that inner product is an effective approach to measure the vector similarity, having been widely used in classic CF models <cit.>. As such, many neural network take it for granted to use element-wise product, such as JRL <cit.> and NeuMF <cit.>. 


Concatenation appends the elements in item embedding to the user embedding vector: 

    ℱ(f^U(u),f^I(i)) = 
    [f^U(u)_1, ..., f^U(u)_K, f^I(i)_1, ..., f^I(i)_K]^T

This operation keeps the raw information in user and item embeddings, without explicitly modeling any interaction between them.  
As such, the model has to expect the following hidden layers to learn the interaction signal in the embeddings.


It is worth highlighting that one main drawback of the two merge functions is that, the embedding dimensions are assumed to be independent. In other words, no correlations among the dimensions are captured — in element-wise product only the interactions of the same dimension are modeled, whereas in concatenation no any interactions are considered. 
Although the following hidden layers might be able to learn the correlations, such process is rather implicit and there is no guarantee that the desired correlations can be successfully captured. One empirical evidence is that the MLP (i.e., concatenation followed by three fully connected hidden layers) even underperforms the simple MF model <cit.>. Another evidence is from  <cit.>, which shows that MLP has to use much more parameters to approximate the inner product well, indicating the limitation in representation ability.





§ CONVNCF FRAMEWORK


This section presents our proposed ConvNCF framework. We first give an overview of the framework, followed by elaborating the key design of outer product and convolutional layers for modeling the embedding dimension correlations. Finally we describe the model prediction and training method.  




 §.§ Framework Overview

We focus on the neural structures since the neural network-based models in recent works always perform well in various learning models <cit.>. Then we select CNN as our fundamental neural structure because, 1) the feature map performs as a 2D matrix, which naturally adapts to the input of CNN, 2) the subregion in the feature map has a spatial relationship, the correlation among multiple dimensions, which could be represented by convolutional filter, and 3) through the convolutional layers, the correlations among all embedding dimensions could be captured from locally to globally. Compared with MLP, the usual neural structure in neural recommendation models, CNN has fewer parameters which lead to less over-fitting. Therefore, we propose the CNN-based framework, ConvNCF.


Figure <ref> illustrates the ConvNCF framework, with the embedding size of 64 and six convolution layers as an example. The embedding layer (left most) contains the embedding functions f^U(u) and f^I(i), which outputs two vectors  (of size 64) to represent user u and item i, respectively. 

Above the embedding layer, ConvNCF models the pairwise correlations between embedding dimensions by constructing the Interaction Map E∈ℝ^64× 64, which is the outer product of user embedding and item embedding. The interaction map is then fed to a stack of convolutional layers to learn high-order correlations; the convolutional layers follow a tower structure with 32 feature maps in the example, and the last convolutional layer outputs a tensor sized 1× 1 × 32. In the prediction layer, ConvNCF applies a linear projection on the 1× 1 × 32 tensor to obtain the prediction ŷ_ui. Given the model prediction, ConvNCF is trained with the pairwise learning objective BPR (Bayesian Personalized Ranking <cit.>).





 §.§ Outer Product Layer

We merge user embedding f^U(u)∈ℝ^K× 1 and item embedding f^I(i)∈ℝ^K× 1 with an outer product operation, which results in a matrix E∈ℝ^K × K: 




    E = f^U(u) ⊗ f^I(i) = f^U(u) · f^I(i)^T,

where  the (k_1,k_2)-th element in E is: e_k_1, k_2 = f^U(u)_k_1· f^I(i)_k_2. 
It can be seen that all pairwise embedding dimension correlations are encoded in the  E, thus we term it as Interaction Map. 

Compared to the widely used inner product operation or element-wise product, we argue that interaction map generated via outer product is more advantageous in threefold: 1) it subsumes the element-wise product which considers only diagonal elements in our interaction map; 2) it encodes more signal by accounting for the correlations between different embedding dimensions; and 3) it is more meaningful than the simple concatenation operation, which only retains the original information in embeddings without modeling any correlation. 
Moreover, it has been recently shown that, modeling the interaction of feature embeddings explicitly is particularly useful for a deep learning model to generalize well on sparse data, whereas using concatenation is less effective <cit.>. 

Another potential benefit of the interaction map lies in its 2D matrix format — which is the same as an image. In this respect, the pairwise correlations encoded in our interaction map can be seen as the local features of an “image”. 
As we all know, deep learning methods have achieved the most success in computer vision, and many powerful deep models especially the ones based on CNN (e.g., 

ResNet <cit.> and DenseNet <cit.>) have been developed for learning from 2D image data. Thus, building a 2D interaction map allows these powerful CNN models to be applied in  the recommendation task.



 §.§ Convolutional Layers

The interaction map E is fed into multiple convolutional layers for learning high-order correlations. As Figure <ref> illustrates, there are 6 convolutional layers with the kernel size 2× 2 and stride 2, meaning that each successive layer is half size of the previous layer. Each convolutional layer  

is composed of 32 convolution kernels (note that pooling is not used in this work).

In this structure, higher layers extract higher-order correlations among dimensions. For example, an element in Layer 1 indicates a 2× 2 area in E, while an element in Layer 2 indicates a 4× 4 area in E. The 4× 4 area indicates the correlations among 4 dimensions. Thus the last layer (i.e., the Layer 6 in Figure <ref>) contains the correlations among all the input embedding dimensions. 


There are two reasons that convolutional layers would be effective over the feature map. Firstly, as a local feature extractor, the convolutional filter does not require that all the subregions have the same rule. In image applications, the image patches cropped from the border and the center are much different. Similarly, in text applications, the phrases captured from the beginning, the middle, and the end of the sentences follow different grammar rules. Secondly, the interaction map may have a spatial relationship, where the subregion indicates the correlations between multiple dimensions, as discussed in Section 4.3.3.









  §.§.§ Advantages over MLP
 


Above the interaction map, the choice of neural layers has a large impact on its performance. A straightforward solution is to use the MLP network as proposed in NCF <cit.>; note that to apply MLP on the 2D interaction matrix E∈ℝ^K× K, we can flat E to a vector of size K^2.
Despite that MLP is theoretically guaranteed to have a strong representation ability <cit.>, its main drawback of having a large number of parameters cannot be ignored. 
As an example, assuming we set the embedding size of a ConvNCF model as 64 (i.e., K=64) and follow the common practice of the half-size tower structure.

In this case, even a 1-layer MLP has 8,388,608 (i.e., 4,096× 2,048) parameters, not to mention the use of more layers. We argue that such a large number of parameters makes MLP prohibitive to be used for prediction because of three reasons: 1) It requires powerful machines with large memories to store the model; and 2) It needs a large number of training data to learn the model well; and 3) It needs to be carefully tuned on the regularization of each layer

to ensure good generalization performance[In fact, another empirical evidence is that most papers used MLP with at most 3 hidden layers, and the performance only improves slightly (or even degrades) with more layers <cit.>].

In contrast, convolution filter can be seen as the “locally connected weight matrix” for a layer, since it is shared in generating all entries of the feature maps of the layer. This significantly reduces the number of parameters of a convolutional layer compared to that of a fully connected layer. Specifically, in contrast to the 1-layer MLP that has over 8 millions parameters, the above 6-layer CNN has only about 20 thousands parameters, which are several magnitudes smaller. This not only allows us to build deeper models than MLP easily, but also benefits the stable and generalizable learning of high-order correlations among embedding dimensions.




  §.§.§ Details of Convolution



Each convolution layer has 32 kernels, each of which produces a feature map. A feature map c in convolutional layer l is represented as a 2D matrix E^lc; since we set the stride to 2, the size of E^lc is half of its previous layer l-1, e.g., E^1c∈ℝ^32× 32 and E^2c∈ℝ^16× 16. 
For Layer l, all feature maps together  can be represented as a 3D tensor ℰ^l. 

Given the input interaction map E, we can first get the feature maps of Layer 1 as follows:

    ℰ^1   = [e^1_i,j,c]_32× 32× 32,   where
    
    e^1_i,j,c   = ReLU(b_1 + ∑_a=0^1 ∑_b=0^1
     e_2i+a,2j+b·  t^1_a,b,c_convolutional kernel),

where b_1 denotes the bias term for Layer 1, and 𝒯^1=[t^1_a,b,c]_2
× 2× 32 is a 3D tensor denoting the convolutional kernel for generating feature maps of Layer 1. We use the rectifer unit as activation function, a common choice in CNN to build deep models. 
Following the similar convolution operation, we can get the feature maps for the following layers. The only difference is that from Layer 1 on, the input to the next layer l+1 becomes a 3D tensor ℰ^l:

    ℰ^l+1   = [e^l+1_i,j,c]_s× s× 32,   where  1≤ l≤ 5,  s = 64/2^l+1, 
     e^l+1_i,j,c   = ReLU(b_l+1 + ∑_a=0^1 ∑_b=0^1
     e^l_2i+a,2j+b·t^l+1_a,b,c),

where b_l+1 denotes the bias term for Layer l+1, and  𝒯^l+1=[t^l+1_a,b,c,d]_2× 2× 32× 32 denote the 4D convolutional kernel for Layer l+1. The output of the last layer is a tensor of dimension 1× 1 × 32, which can be seen as a vector and is projected to the final prediction score with a weight vector w. 








  §.§.§ Dimension Correlation in ConvNCF


Here we provide detailed explanations on how ConvNCF models the dimension correlations between embedding factors. Equation <ref> demonstrates that the entry e_xy in the interaction map E is a product of f^U(u)_x and f^I(i)_y, which is viewed as a 1-to-1 correlation, also namely 1-order correlation or basic correlation. Let [xs:xe] be a row range and [ys:ye] be a column range, the entries in the adjacent subregion E_xs:xe,ys:ye indicate all the basic correlations between f^U(u)_xs:xe and f^I(i)_ys:ye. Overall, the interaction map E itself contains all the basic correlations between f^U(u) and f^I(i). Note that, the first layer of ConvNCF is the interaction map E, which means what ConvNCF actually does is to predict a score by modeling all the basic correlations.

Then we elaborate on how ConvNCF models the correlations. Equation <ref>  
demonstrates that the feature e^1_xy is the composite correlation of four entries in the interaction map E, [e_2x,2y; e_2x,2y+1; e_2x+1,2y; e_2x+1, 2y+1]. Thus the entry e^1_xy in the the feature map E^1 is actually the feature of the composite correlations of  E_2x:2x+1,2y:2y+1, which is a 2-to-2 correlation namely 2-order correlation. Therefore, the feature map E^1 is composed of 2-order correlations. Similarly, above the feature map E^1, Equation <ref> demonstrates that the entry e^2_xy in E^2 is the composite correlations of E^1_2x:2x+1,2y:2y+1, which is the 4-to-4 composite correlations of E_4x:4x+3,4y:4y+3 namely 4-order correlation. Thus, the feature map E^2 is composed of 4-order correlations.
It is worth mentioning that the convolutional kernels are sized 2×2 with stride 2 and no padding. The entries in higher feature map can just cover all the entries in lower feature map. This ensures the integrity in transferring the embedding information. As such, an entry in the last hidden layer encodes the correlations among all dimensions. Through this way of stacking multiple convolutional layers, we allow ConvNCF to learn high-order correlations among embedding dimensions from locally to globally, based on the 2D interaction map. 







 §.§ Model Prediction and Training


Given the output vector of the last convolutional layer as g∈ℝ^K× 1, the model prediction is defined as: ŷ_ui = w^Tg, where w is the trainable weight vector in the prediction layer. 

To summarize, the parameters in ConvNCF are grouped into four parts, Θ_U in user embedding function f^U(·), Θ_I in item embedding generation f^I(·), Θ_CNN in the convolutional layers, and w for the prediction layer. 


Considering that recommendation is a personalized ranking task, we train the parameters with a ranking-aware objective function. Here we adopt the Bayesian Personalized Ranking (BPR) objective function <cit.>:

    L    = ∑_(u,i,j)∈𝒟 -lnσ(ŷ_ui - ŷ_uj)
       + λ_1 ||Θ_U||^2 + λ_2 ||Θ_I||^2 + λ_3 ||Θ_CNN||^2 +  λ_4 ||w||^2 ,

where λ_* are hyper-parameters for regularization, and 𝒟 denotes the set of training instances: 𝒟:= {(u,i,j)| i∈ℛ_u^+ ∧ j∉ℛ_u^+ }, where ℛ_u^+ denotes the set of items that has been consumed by user u.

In each training epoch, we first shuffle all observed interactions, and then 
get a mini-batch in a sequential way. 
Given the mini-batch of observed interactions, we generate negative examples on the fly to get the training triplets. The negative examples are randomly sampled from a uniform distribution; while recent efforts show that a better negative sampler can further improve the performance <cit.>, we leave this exploration as future work. 


§ THREE CONVNCF METHODS


To demonstrate how ConvNCF works, we propose three instantiations of ConvNCF by equipping them with different embedding functions. We name the three methods as ConvNCF-MF, ConvNCF-FISM and ConvNCF-SVD++, which adopt the embedding function used in matrix factorization (MF) <cit.>, factored item similarity model (FISM) <cit.>, and SVD++ <cit.>, respectively. Figure <ref> illustrates the three embedding mechanisms, which differ in the user embedding function part (the item embedding functions are the same, which project item ID to embedding vector). Then the embedding vectors are fed into the merge function ℱ (the outer product) to generate the interaction map E, which is fed into the ConvNCF subsequently. Next, we present the three methods in turn; and lastly, we discuss the important tricks of embedding pre-training and adaptive regularization, which are crucial for the effectiveness of ConvNCF methods. 



 §.§ ConvNCF-MF





MF adopts the simplest ID embedding function, i.e., directly projecting the ID of a user (an item) to latent vector representation:

    f^U(u) = p_u,   
    f^I(i) = q_i.

Let the user embedding matrix be P∈ℝ^M× K and the item embedding matrix be Q∈ℝ^N× K. This function is equal to taking out the u-th row of P as u's embedding vector (i.e., p_u) and the i-th row of Q as i's embedding vector (i.e., q_i). 

In MF, each user is associated with an individualized set of parameters to denote the user's interest — even two users have exactly the same interactions on items, they are parameterized differently in the model. As such, it is also called as user-based CF in recommendation literature. The problem of this setting is that, if a user has never been seen in the training set (i.e., cold-start), the model cannot obtain her embedding in the online phase, even though the user has some new interactions on items. In other words, the model lacks the ability to provide real-time personalization. Next, we introduce the setting of item-based CF, we can address this deficiency. 









 §.§ ConvNCF-FISM

Instead of directly projecting user ID to embedding vector, FISM represents a user with her historically interacted items, based on which the embedding function is defined:


    f^U(u) = 1/|ℛ_u|^α∑_t∈ℛ_u\{i}q'_t,   
        f^I(i) = q_i,

where ℛ_u denotes the set of items interacted by u, q'_t denotes the embedding of historical item t in constructing the user embedding function. Note that FISM has three considerations to ensure the embedding quality: 1) the historical item embedding q'_t used in user embedding function is different from target item embedding q_t used in item embedding function, which can improve the model representation ability; 2) the target item i is excluded to represent the user, i.e., the ℛ_u\{i} term, so as to avoid information leakage in training; 3) the coefficient 1/|ℛ_u|^α is to normalize users of different activity levels (by convention we set α as 0.5). 

In FISM, a user's embedding is aggregated from the embeddings of the user's historically interacted items. Thus, for two users with the same interactions on items, their embeddings are the same. For a cold-start user, as long as she has new interactions, we can obtain her embedding instantaneously by calling the embedding function without re-training the model. Therefore, such item-based CF scheme is suitable to provide real-time personalization to refresh the recommendation for users that have new interactions.  





 §.§ ConvNCF-SVD++

SVD++ is a hybrid method that combines the user embedding design of MF and FISM. It uses both the ID and historically interacted items to represent a user, and defines the embedding function as:


    f^U(u) = p_u + 1/|ℛ_u|^α∑_t∈ℛ_u\{i}q'_t,    
        f^I(i) = q_i,

where the notations follow the ones used in Equation (<ref>) and (<ref>). The user embedding function of SVD++ is the sum of the user embedding functions of MF and FISM, unifing the strengths of both methods. Specifically, p_u is a static vector to encode user inherent preference, and ∑_t∈ℛ_u\{i}q'_t can be dynamically adjusted by including recently interacted items. Note that SVD++ is a very competitive CF model that is known as the best single model in the Netflix Challenge <cit.>. By plugging its embedding functions into ConvNCF, we can further advance its performance. 






 §.§ Embedding Pre-training and Adaptive Regularization

Due to the multi-layer design of ConvNCF, the initialization of model parameters has a large impact on model training and its testing performance. As the convolutional layers are learned on the outer product of embeddings to generate the features used for prediction, a good initialization on embeddings is beneficial to the overall model learning. Thus, we propose to pre-train the embedding layer. For the three ConvNCF methods, we first train its shallow model counterpart, i.e., MF, FISM, and SVD++ for ConvNCF-MF, ConvNCF-FISM, and ConvNCF-SVD++, respectively. We then use the learned embeddings to initialize the embedding layer of the corresponding method. Our empirical studies find that this strategy can substantially improve the model performance. 


After pre-training the embedding layer, we start training the ConvNCF model. As other model parameters are randomly initialized, the overall model is in an underfitting state. Thus, we disable regularization for the 1st epoch to make the model learn to fit the data as quickly as possible. For the following epochs, we enforce regularization on ConvNCF to prevent overfitting, including the L_2 regularization on the embedding layer (controlled by λ_1 and λ_2), convolution layers (controlled by λ_3), and the output layer (controlled by λ_4). It is worth noting that the regularization coefficients (especially λ_4 on the output layer) have a very large impact and should be carefully tuned for an optimal performance. 

























§ EXPERIMENTS


To evaluate our proposal, we conduct experiments to answer the following research questions:

	
  RQ1 How do our ConvNCF perform compared with state-of-the-art recommendation methods?
	
  RQ2 How do the embedding dimension correlations (captured by the outer product layer and convolution layers) contribute to the model performance? 


	
  RQ3 How do the key hyper-parameters (e.g., feature maps and pre-training) affect ConvNCF's performance?




 §.§ Experimental Settings




  
Data Description
We conduct experiments on two publicly accessible datasets: Yelp[https://github.com/hexiangnan/sigir16-eals] and Gowalla[http://dawenl.github.io/data/gowalla_pro.zip].

Yelp. This is the Yelp Challenge data for user ratings on businesses.
We filter the dataset same as <cit.>. Moreover, we merge the repetitive ratings at different timestamps to the earliest one, so as to study the performance of recommending novel items to a user.
The final dataset obtains 25,815 users, 25,677 items, and 730,791 ratings.

Gowalla. This is the check-in dataset from Gowalla, a location-based social network, constructed by <cit.> for item recommendation.

To ensure the quality of the dataset, we perform a modest filtering on the data, retaining users with at least two interactions and items with at least ten interactions.
The final dataset contains 54,156 users, 52,400 items, and 1,249,703 interactions. 



  
Evaluation Protocols
For each user in the dataset, we holdout the latest one interaction as the testing positive sample, and then pair it with 999 items that the user did not rate before as the negative samples.
Each method then generates predictions for these 1,000 user-item pairs.
To evaluate the results, we adopt two metrics Hit Ratio (HR) and Normalized Discounted Cumulative Gain (NDCG), same as  <cit.>.
HR@k is a recall-based metric, measuring whether the testing item is in the top-k position (1 for yes and 0 otherwise).
NDCG@k assigns higher scores to the items within the top k positions of the ranking list.
To eliminate the effect of random oscillation, we report the average scores of the last ten epochs upon convergence. 



  
Baselines
To justify the effectiveness of our method, we compare with the following methods:



  ItemPop ranks the items based on their popularity, which is calculated by the number of interactions. It is always taken as a benchmark for recommender algorithms.


  MF-BPR <cit.> optimizes the matrix factorization model with the pairwise BPR ranking loss. It is a competitive user-based CF method.  


  FISM <cit.> replaces the user ID embedding with the item-based embedding function of Equation <ref>. 
It is a competitive item-based CF method.


  SVD++ <cit.> combines the user embedding design of MF and FISM, as formulated in <ref>. It is a strong CF model that scores the best single model in the Netflix challenge. 





  MLP <cit.> is a NCF method that feeds the concatenation of user embedding and item embedding into the standard MLP for learning the interaction function. As no interaction between user embedding and item embedding is explicitly modeled, this model can be inferior to the MF model <cit.>. 


  JRL <cit.> is a NCF method that places a MLP above the element-wise product of user embedding and item embedding. It enhances GMF <cit.> by placing multiple hidden layers above the element-wise product, while GMF directly outputs the prediction score from the element-wise product. 


  NeuMF <cit.> is a state-of-the-art method for item recommendation, which ensembles GMF and MLP to learn the user-item interaction function.


The architecture of ConvNCF is shown in Figure <ref>. Above the outer product layer, there are six convolutional layers. Each of them has 32 convolutional kernels with stride 2 and no padding.
We implement the three ConvNCF methods as introduced in Section <ref>, which have the same architecture setting with difference only in the user embedding function. 




  
Parameter Settings
Our methods are implemented in Tensorflow[Available at: https://github.com/duxy-me/ConvNCFhttps://github.com/duxy-me/ConvNCF.].  We randomly holdout 1 training interaction for each user as the validation set to tune hyper-parameters.
For a fair comparison, all models apply the same setting in terms of the model size and optimization: the embedding size is set to 64 and all models are optimized with the BPR loss using mini-batch Adagrad. 






For MLP, JRL, and NeuMF that have multiple fully connected layers, we tuned the number of layers from 1 to 3 following the tower structure of <cit.>. 
For all models besides MF, we pre-train their embedding layers using MF embeddings, and the L_2 regularization for each method has been fairly tuned. 


Due to the strong representation ability of ConvNCF, it is prone to overfitting. As such, tuning the regularization has a large impact on its performance. We divide the parameters of ConvNCF into two groups: embedding parameters (Θ_U and Θ_I) and CNN parameters (Θ_CNN and w), and tune the regularization coefficient and learning rate separately for the two groups. For the embedding parameters, the optimal learning rate (under Adagrad) is around 0.001 and 0.005, and the regularization coefficients (i.e., λ_1, λ_2) are searched in [0,10^-7,10^-6]. 
For the CNN parameters, the optimal learning rate is around 0.01 and 0.05, and the regularization coefficients (i.e., λ_3, λ_4) are searched in [10^-2,10^-1,1,10]. 





 §.§ Performance Comparison (RQ1)

	Table <ref> shows the Top-k recommendation performance on both datasets where k is set to 5, 10, and 20.
	We have the following key observations:
	
	    
  * Our ConvNCF methods achieve the best performance in general, and obtain high improvements over the compared methods. Particularly, each ConvNCF instantiation outperforms its linear counterpart in all settings (e.g., ConvNCF-MF vs. MF, ConvNCF-FISM vs. FISM). 
	    This justifies the utility of our ConvNCF design — using outer product (and CNN) to capture second-order (and high-order) correlations among embedding dimensions. 
	    
		
  * Among the three ConvNCF methods, ConvNCF-SVD++ is the strongest and achieves the best performance in most settings. This demonstrates the utility of designing better embedding function for ConvNCF. Moreover, we find that ConvNCF-FISM achieves weak performance on Gowalla. We hypothesize the reason is that the item-based user representation mechanism does not work well for the Gowalla dataset, since FISM also underperforms MF substantially on Gowalla. Another obsrevation is that initialized with the pretrained embeddings, ConvNCF would significantly improve the performances in few epochs (Figure <ref>). In such few epochs, the embeddings would not change much, that implies that modeling the embedding correlations plays a significant role for the state-of-the-art performances.  
		
 	    
		
  * JRL consistently outperforms MLP by a large margin on both datasets.
		This indicates that, explicitly modeling the correlations of embedding dimensions is rather helpful for the followup learning of the hidden layers, even modeling the simple correlations that assume the dimensions are independent. 
		Meanwhile, it also reveals the practical difficulties to train MLP well, although it has strong representation ability in principle <cit.>.
		
  * The relatively weak performances of MF and FISM reflect that simple inner product of the user and item embeddings is far insufficient to depict the complex patterns within the user-item interactions. Moreover, MF outperforms FISM on Gowalla but underperforms FISM on Yelp. This implies that which embedding function works better is dependent on the properties of the dataset.
		
		Lastly, ItemPop achieves the worst performance, verifying the importance of considering users' personalized preferences in the recommendation.
	
	
	

	

 §.§ Modeling Embedding Dimension Correlations (RQ2)

	
	
	
	Through overall performance comparison, we have shown the strength of ConvNCF. Next, we conduct more experiments to verify the utility of modeling embedding dimension correlations, more concretely, the efficacy of outer product and CNN in ConvNCF. We choose ConvNCF-MF as an example to demonstrate the usefulness of outer product (for brevity, we term it as ConvNCF in this subsection only). 

	
	
	

  §.§.§ Efficacy of Outer Product

	Besides outer product, another two choices that are commonly used in previous work are concatenation and element-wise product.
	It is worth mentioning that element-wise product, concatenation, and outer product operations have essentially different structures, since the outer product operation is performed based on a matrix while the other two operations are performed based on vectors. To conduct a fair evaluation, we implement the state-of-the-art models representing element-wise product and the concatenation style learning methods according to  <cit.>.
	As such, we compare the training progress of ConvNCF with GMF and JRL (which use element-wise product), and MLP (which uses concatenation). 
	As can be seen from Figure <ref>, ConvNCF outperforms other methods by a large margin on both datasets, verifying the positive effect of using outer product above the embedding layer. 
	Specifically, the improvements over GMF and JRL demonstrate that explicitly modeling the correlations between different embedding dimensions are useful.
	Lastly, the rather weak and unstable performance of MLP imply the difficulties to train MLP well, especially when the low-level has fewer semantics about the feature interactions. This is consistent with the recent finding of <cit.> in using MLP for sparse data prediction. 
	
	
 
		
		
		
		
	
	
	
	
	

  §.§.§ Efficacy of CNN
 	In order to verify the effectiveness of CNN over MLP, we make a fair comparison by training them based on the same 2D interaction map. 
	Specifically, we first flatten the interaction map as a K^2 dimensional vector, and then place a 3-layer MLP above it. We term this method as ONCF-mlp.  Figure <ref> compares its performance with ConvNCF in each epoch. We can see that ONCF-mlp performs much worse than ConvNCF, in spite of the fact that it uses much more parameters (1000 times more) than ConvNCF. Another drawback of using such many parameters in ONCF-mlp is that it makes the model rather unstable, which is evidenced by its large variance in epoch. 
	In contrast, our ConvNCF achieves much better and stable performance by using the locally connected CNN. These empirical evidence provide support for our motivation of designing ConvNCF and our discussion of MLP's drawbacks in Section <ref>.
	
	
	
	
		
		
	
	
	
	

 §.§ Hyperparameter Study (RQ3)

	
	
	

  §.§.§ Impact of Feature Map Number
 The number of feature maps in each CNN layer affects the representation ability of our ConvNCF. 
	
	Figure <ref> shows the performance of ConvNCF-MF with respect to different numbers of feature maps.
	
	We can see that all the curves increase steadily and finally achieve similar performance, though there are some slight differences on the convergence curve. This reflects the strong expressiveness and generalization of using CNN above the interaction map since dramatically increasing the number of parameters of a neural network does not lead to overfitting. Consequently, our model is very suitable for practical use.


	
	















	





  §.§.§ Pretraining on Embeddings.
 In our experiments, all the embeddings used in ConvNCF are initialized by the pretrained parameters. We here compare the effect of training with pretraining and training from scratch. As shown in Figure <ref>, there are two curves. The orange one indicates the performance training with pretraining, and the purple one indicates the performance training from scratch. The orange curve at the left side of the dashed line presents the status pretraining the models and the rest presents the status training ConvNCF. Due to the simplicity of original models, the pretraining processs increase the performances soon. Based on these embeddings, ConvNCF obtains the state-of-the-art performance, which re-emphasizes the importance of modeling the dimension correlations. In contrast, training from scratch also grasps some information but due to more complex structure, it requires more efforts to achieve the nice performance.







§ CONCLUSION

In this paper, we presented a new neural network framework ConvNCF for collaborative filtering by modeling  the embedding dimension correlations. Moreover, we  proposed three variants of ConvNCF.

ConvNCF is able to capture latent relations between embedding dimensions via the outer product operation. 

To learn more accurate preference, we used multiple convolution layers on top of the interaction map. Extensive experiments on two real-world datasets demonstrated that modeling the embedding dimension correlations was helpful for CF tasks. We further showed that ConvNCF outperforms state-of-the-art methods in the top-k item recommendation task. 



In future, we will explore more advanced CNN techniques such as attentive mechanism <cit.> and residual learning <cit.> to learn higher-level representations.
Moreover, we will extend ConvNCF to content-based recommendation scenarios <cit.> since the item features have richer semantics than just an ID. Particularly, we are interested in building recommender systems for multimedia items like images and videos, and textual items like news.






This work was partially supported by the National Key Research and Development Program of China under Grant 2016YFB1001001, the National Natural Science Foundation of China (Grant No. 61772275, 61732007, 61720106004 and U1611461), the Natural Science Foundation of Jiangsu Province (BK20170033), and the National Science Foundation of China - Guangdong Joint Foundation (Grant No. U1401257). This research is also part of NExT++, supported by the National Research Foundation, Prime Ministers Office, Singapore under its IRC@Singapore Funding Initiative.




ACM-Reference-Format



