MMGRec: Multimodal Generative Recommendation with Transformer Model
Han Liu Yinwei Wei Xuemeng Song Weili Guan Yuan-Fang Li Liqiang Nie

















  Shandong University
  
  
  China
  
hanliu.sdu@gmail.com




  Monash University
  
  
  Australia
weiyinwei@hotmail.com



  Shandong University
  
  
  China
sxmustc@gmail.com



  Harbin Institute of Technology (Shenzhen)
  
  
  China
honeyguan@gmail.com



  Monash University
  
  
  Australia
yuanfang.li@monash.edu



  Harbin Institute of Technology (Shenzhen)	
  
  
  China
nieliqiang@gmail.com












Multimodal recommendation aims to recommend user-preferred candidates based on her/his historically interacted items and associated multimodal information. Previous studies commonly employ an embed-and-retrieve paradigm: learning user and item representations in the same embedding space, then retrieving similar candidate items for a user via embedding inner product. However, this paradigm suffers from inference cost, interaction modeling, and false-negative issues. Toward this end, we propose a new MMGRec model to introduce a generative paradigm into multimodal recommendation. Specifically, we first devise a hierarchical quantization method Graph RQ-VAE to assign Rec-ID for each item from its multimodal and CF information. Consisting of a tuple of semantically meaningful tokens, Rec-ID serves as the unique identifier of each item. Afterward, we train a Transformer-based recommender to generate the Rec-IDs of user-preferred items based on historical interaction sequences. The generative paradigm is qualified since this model systematically predicts the tuple of tokens identifying the recommended item in an autoregressive manner. Moreover, a relation-aware self-attention mechanism is devised for the Transformer to handle non-sequential interaction sequences, which explores the element pairwise relation to replace absolute positional encoding. Extensive experiments evaluate MMGRec's effectiveness compared with state-of-the-art methods.





















































MMGRec: Multimodal Generative Recommendation with Transformer Model
    Liqiang Nie
    April 29, 2024
===================================================================



§ INTRODUCTION

With the popularity of multimedia-centric scenarios like social media and micro-video websites, multimodal recommendation systems are gaining widespread attention and adoption. These systems generally follow the embed-and-retrieve paradigm. The paradigm first incorporates multimodal information with collaborative filtering (CF) <cit.> to learn representations for users and items in the embedding stage, and then locates the preferred items for users by measuring inner product similarity in the retrieval stage. Relevant research primarily focuses on enhancing representation learning in the embedding stage. For example, ACF <cit.> introduces a hierarchical attention mechanism to select informative modality content, while MMGCN <cit.> incorporates Graph Convolution Network (GCN) to achieve modal information aggregation and propagation.

Despite the remarkable performance, existing research overlooks the inherent issues caused by similarity calculation in the retrieval stage. First, similarity calculation becomes overwhelming. With I items and U users, capturing the top similar K items for each user incurs time complexity 𝒪(UID+UIlogK), where D is the dimension of user/item representations. As users/items increase, the time consumption significantly grows, affecting recommendation efficiency. Secondly, linear inner product inadequately models user-item interactions' intricate structure <cit.>. Although some studies model non-linear interaction with neural networks <cit.> and metric learning <cit.>, they sacrifice similarity calculation speed, exacerbating efficiency issues. Lastly, the paradigm assumes that interacted items are inherently closer to user preference than uninteracted ones <cit.>, leading to a false-negative issue <cit.> since the absence of interaction does not necessarily imply dislike <cit.>. 


To address the issues, recent work <cit.> inherits the Differentiable Search Index algorithm <cit.> and presents a generative paradigm-based recommendation method, which aims to directly generate the list of user-preferred items without similarity calculation. To achieve this goal, two components, including Semantic ID Assignment and Semantic ID Generation, are designed to answer the questions: how to represent an item with a semantic ID and how to predict the semantic IDs of preferred items for a user. Initially, the method assigns each item a unique semantic ID — a sequence of semantic tokens learned from its textual description. Then, a Transformer <cit.> model is trained as the recommendation agent to predict the semantic ID of the next item based on the user's chronological interaction sequence. Each ID token is yielded with autoregressive decoding, and multiple item IDs are generated with beam search <cit.>. However, for multimodal recommendation, there are still two technical challenges untouched in the two components:


  * Semantic ID Assignment: Existing ID assignment approaches employ unimodal semantic clustering, inevitably leading to ID collision among similar items. Although random distinction <cit.> is introduced to address this, such randomness information in ID is difficult to learn with Transformer, negatively affecting the performance of generative recommendation.


  * Semantic ID Generation: The position and order information in historical interaction sequences is important for Transformer to understand user behavior and preference. However, this information often lacks explicit references (e.g., timestamp, rating) and varies across different users. Therefore, it is challenging to make the Transformer aware of each item's position within the historical interaction sequence.


To overcome the challenges, we design a new ID structure — Rec-ID, and present a Multimodal Generative Recommendation (MMGRec) framework for Rec-ID assignment and generation. Specifically, 

Rec-ID comprises a sequence of semantic tokens followed by a popularity token, leveraging the correlation between item popularity and semantic information.
For Rec-ID assignment, we devise a Graph Residual-Quantized Variational AutoEncoder (Graph RQ-VAE) to fuse item multimodal information with CF information and quantize them into a sequence of semantic tokens. Then, we rank the items with identical semantic tokens by popularity and use their ranking index as the last token to avoid the collision problem. After that, for Rec-ID generation, we train a Transformer-based model to directly generate the preferred items' Rec-IDs based on users' historical interactions. To capture item position information in historical interaction sequences, we design a relation-aware self-attention mechanism for Transformer. By extending the original self-attention, this mechanism constructs user-specific relations among items under the supervision of user parameters. We conduct extensive experiments on three public datasets to demonstrate the rationality and effectiveness of the proposed MMGRec.
  

Overall, the main contributions of our work are three-fold:


  * We propose MMGRec, a novel Transformer-based recommendation framework that consists of Rec-ID assignment and generation. This is the first effort known to us to introduce the generative paradigm into multimodal recommendation.

  * Technically, we design a multimodal information quantization algorithm Graph RQ-VAE for Rec-ID assignment, and a relation-aware self-attention mechanism within Transformer for Rec-ID generation.


  * We conduct empirical studies on three real-world datasets. Extensive results demonstrate that MMGRec achieves state-of-the-art performance with promising inference efficiency. 




§ PRELIMINARY






This work is inspired by TIGER (short for Transformer Index for GEnerative Recommenders), a recent generative model for sequential recommendation <cit.>. Its foundation lies in a novel semantic ID for item identification, consisting of an ordered semantic token tuple:

    (c_1, ⋯, c_M-1)=ϕ(𝐟^t),

where 𝐟^t is the item's textual feature converted from descriptions using Sentence-T5 <cit.>. ϕ(·) indicates the vector quantization algorithm <cit.> converting 𝐟^t into a tuple of M-1 semantic tokens.

To address the collision problem that items with similar textual features tend to be assigned the same tokens,  the method inserts an extra token at the end of the semantic tokens to ensure the uniqueness of IDs. Specifically, given K items with the same tokens (c_1, ⋯, c_M-1), it randomly sorts the items and sets their corresponding indices in the sorted order as the extra tokens:

    (p_1, p_2, ⋯, p_K)= argsort_rand(item_1, item_2, ⋯, item_K),

where argsort_rand(·) is the random ranking function to sort K items. p_k∈{1, 2, ⋯, K} denotes the index of the k-th item in the order. Then, after concatenating them at the end of semantic tokens, the semantic ID of item k can be obtained as:

    S_k = (c_1, ⋯, c_M-1, p_k).






With the obtained semantic ID, a Transformer-based sequence-to-sequence model is trained on the sequences of semantic IDs associated with items from a user's chronological interactions. The objective is to predict the semantic ID of the next item after the sequence. This model autoregressively decodes the tokens of semantic ID identifying the next item,  thereby qualifying as a generative recommendation. 



However, this method has notable limitations. The random token lacks semantic meaning and is challenging to model using statistical machine learning techniques like Transformer. Additionally, the method struggles with the input of non-sequential interactions.






































§ METHOD


In this section, we first elaborate on our designed MMGRec model according to its two components — Rec-ID assignment and Rec-ID generation. After that, we detail the model training and inference, followed by the discussion regarding the time complexity.




 §.§ Rec-ID Definition
  
As the foundation of generative recommendation, we define a new item identifier Rec-ID equipped with two attributes:
1) Semantics, where the Rec-ID condenses the semantic information of the item, and 2) Uniqueness, where the Rec-ID is capable of distinguishing each item from others. 

Towards this end, Rec-ID is designed to consist of a sequence of semantic tokens and an extra token based on the item's popularity. The motivation is that the popularity of an item is relevant to its semantic information. Hence, during the autoregressive generation phase, the last token (i.e., popularity token) can be predicted by jointly analyzing the semantic tokens generated before.



 §.§ Rec-ID Assignment
 
According to the design of Rec-ID, we detail the Rec-ID assignment to describe how to allocate the semantic tokens and the popularity token for each item.


  §.§.§ Graph RQ-VAE

To learn the semantic tokens for items, as shown in Figure <ref>, we devise a Graph RQ-VAE model to fuse their multimodal information and further quantize into codewords. Alongside multimodal (e.g., visual, acoustic, and textual) features, historical user-item interactions are fed to this model. These interactions not only aid in extracting relevant multimodal features for recommendation <cit.> but also reveal behavioral signals related to item popularity. In particular, a user-item bipartite graph is constructed, where nodes represent users and items, and edges denote interactions. To fuse these heterogeneous inputs into a unified representation, we initialize item i as follows:



    𝐡_i^(0)=[ 𝐡_i^m 𝐡_i^c ], 𝐡_i^m=[ 𝐟_i^v 𝐟_i^a 𝐟_i^t ]𝐖,

where 𝐡_i^m∈ℝ^D represents item i's multimodal embedding learned by concatenating its visual, acoustic, and textual features, i.e., 𝐟_i^v∈ℝ^D^v, 𝐟_i^a∈ℝ^D^a, and 𝐟_i^t∈ℝ^D^t. D^v, D^a, and D^t denote the dimensions of these modality features. 𝐖∈ℝ^(D^v+D^a+D^t) × D is a parameter matrix that maps multimodal features to the D-dimensional embedding space. Combining this multimodal embedding with a randomly initialized CF embedding 𝐡_i^c∈ℝ^D yields the item representation 𝐡_i^(0)∈ℝ^2D. For performing graph convolution operations on the bipartite graph, user representations are randomly initialized, such as 𝐡_u^(0)∈ℝ^2D for user u.  

Treating them as node embeddings at the 0-th layer, item i’s representation at the n-th GCN layer is obtained as:

    𝐡_i^(n)=LeakyReLU(𝐡_i^(n-1)𝐖_1 + 1/|𝒩_i|∑_u∈𝒩_i𝐡_u^(n-1)𝐖_2),

where 𝒩_i is the neighborhood set of item i (i.e., users whom item i directly interacted with), and LeakyReLU is the activation function. Analogously, the representation 𝐡_u^(n) for user u is obtained by information propagation from its neighbor items. After stacked GCN layers, final representations are denoted as 𝐡_u∈ℝ^D and 𝐡_i∈ℝ^D. Note that we utilize the concise graph convolution operation <cit.>, leaving complicated choices <cit.> for further exploration.



  


Based on the final item representations, we employ Residual-Quantized Variational AutoEncoder (RQ-VAE) <cit.> to generate semantic tokens of Rec-IDs. As a multi-stage vector quantizer, RQ-VAE can produce a tuple of codewords by quantizing residuals across hierarchical levels. As illustrated in Figure <ref>, we input the learned item representation into an encoder model:  

    𝐳_i = ℰ(𝐡_i),

where 𝐳_i is the output latent vector and ℰ is the encoder, comprising a two-layer Multi-Layer Perceptron (MLP). At the first level, the initial residual is 𝐫_i,1=𝐳_i. A codebook 𝐁_1={𝐛_l}_l=1^L is employed, where 𝐛_l denotes the parameter embedding and L represents the codebook size. Then, 𝐫_i,1 is quantized by mapping it to the closest embedding from this codebook:

    c_i,1=argmin_l||𝐫_i,1-𝐛_l||_2^2,

where c_i,1 is the index of the closest embedding, i.e., the first codeword. Recursively, for subsequent level m, the residual is defined as 𝐫_i,m=𝐫_i,m-1-𝐛_c_i,m-1, and the codeword c_i, m is similarly computed using the level-specific codebook 𝐁_m. By iteratively performing M-1 operations, a sequence of codewords is generated as follows:

    (c_i, 1, ⋯, c_i, M-1)=RQ-VAE(𝐡_i),

where c_i, m denotes the m-th codeword used as the m-th semantic token in Rec-ID of item i. Notably, the recursive approach approximates the input in a coarse-to-fine manner. Since each codeword is sourced from a distinct codebook, the capacity of Rec-IDs to uniquely represent items equals the product of all codebook sizes.



Upon the codewords, we use a two-layer MLP decoder 𝒟 to reconstruct the input 𝐡_i as follows:

    𝐡_i=𝒟(𝐳_i+sg(𝐳_i-𝐳_i)),  𝐳_i=∑_m=1^M-1𝐛_c_i,m,

where 𝐡_i denotes the decoder output, and 𝐳_i is the quantized representation of 𝐳_i. sg(·) represents the stop-gradient operation. The loss function for training Graph RQ-VAE is as follows:


    ℒ=ℒ_bpr   +ℒ_rqvae,   ℒ_bpr=∑_(u,i,j)∈ℛ -ln σ(𝐡_u𝐡_i^⊤-𝐡_u𝐡_j^⊤),
    ℒ_rqvae=   ∑_i=1^I||𝐡_i-𝐡_i||^2 +
        ∑_m=1^M-1||sg(𝐫_i,m)-𝐛_c_i,m||^2 
        + β||𝐫_i,m-sg(𝐛_c_i,m)||^2.

Here ℒ_bpr is the pairwise BPR <cit.> loss to learn user and item representations. ℛ={(u,i,j)|(u,i)∈ℛ^+,(u,j)∈ℛ^-}, where ℛ^+ indicates the observed interactions, and ℛ^- indicates the unobserved interactions. σ(·) is the Sigmoid function. The loss function ℒ_rqvae facilitates the simultaneous training of the encoder-decoder and codebooks in RQ-VAE. I denotes the number of items. We alternatively optimize one of two sub-losses to learn model parameters.



  §.§.§ Handling Collisions



To handle the collision problem, we expand a popularity token at the end of the semantic tokens to ensure Rec-IDs' uniqueness. In particular, given K items with the same tokens (c_1, ⋯, c_M-1), we sort the items according to their popularity, i.e., the total number of interactions with users. The corresponding indices in the sorted order are used as their popularity tokens, formally, 

    (p_1, p_2, ⋯, p_K)= argsort_pop(item_1, item_2, ⋯, item_K),

where argsort_pop(·) is the popularity ranking function to sort K items. p_k∈{1, 2, ⋯, K} denotes the index of the k-th item in the order. Then, after concatenating them at the end of semantic tokens, the Rec-ID of item k can be obtained as: 

    R_k = (c_1, ⋯, c_M-1, p_k).












 §.§ Rec-ID Generation



  §.§.§ Transformer Input Embedding

To implement generative recommendation, a sequence-to-sequence Transformer model is exploited as a recommender. As shown in Figure <ref>, the model consists of stacked encoder and decoder layers. To adopt the model in recommendation tasks, we organize historically interacted items of each user as a sequence of tokens and feed them into the encoder. 

Taking user u as an example, we gather her/his interacted items' atomic IDs together as the Transformer's input, and utilize the learned item representations for input embedding, formally, 

    [ 𝐞_1,⋯,𝐞_N ]=[ 𝐡_i_1,⋯,𝐡_i_N ],




where 𝐞_i∈ℝ^D denotes the embedding of the corresponding item in the sequence. 

It is worth noting that the input sequence length is uniformly set to N, filling the missing positions with padding token 0. 



  §.§.§ Relation-Aware Self-Attention

Following input embedding, positional encoding in Transformer is crucial for providing order information of the input sequence. Unlike sequential recommendation tasks, the input interacted item sequence is not well-ordered in our case, and its actual order implicitly and dynamically changes with different users. Thus, traditional absolute positional encoding is ineffective in this context. To overcome the problem, we develop a relation-aware self-attention mechanism for the Transformer. This mechanism explicitly models user-specific pairwise relations among input items based on user preferences, thereby encoding relative position information within interaction sequences.








Specifically, as shown in Figure <ref>, we extend the self-attention sublayer by incorporating user-specific parameters. First, we introduce the user-specific position encoder, which aims to encode the position information of each item from different users' perspectives:

    𝐩_j^u=𝐞_j𝐖_u^V,

where 𝐖_u^V represents the parameter matrix of user u's position encoder and 𝐩_j^u is the encoded vector of item j. 

After obtaining the encoded vectors, we conduct the multi-head attention operations on the input sequence. In each head, we map the input element embeddings [ 𝐞_1,⋯,𝐞_N ] into query, key, and value spaces, and compute their new embeddings [ 𝐱_1,⋯,𝐱_N ] where 𝐱_i∈ℝ^D: 




    𝐱_i = ∑_j=1^Nα_ij(𝐞_j𝐖^V + 𝐩_j^u),

where 𝐖^V is the matrix mapping the elements to the value space. α_ij is the weight coefficient computed with the Softmax function:







    α_ij = exp(ϵ_ij)/∑_n=1^Nexp(ϵ_in),

wherein ϵ_ij is calculated through a compatibility function that compares two input elements and considers their pairwise relation in the context of user u:

    ϵ_ij = (𝐞_i𝐖^Q)(𝐞_j𝐖^K)^⊤  + (𝐞_i𝐖_u^Q)(𝐞_j𝐖_u^K)^⊤/√(D),

where 𝐖^Q and 𝐖^K are parameter matrices mapping elements to the query and key spaces, respectively. Moreover, elements are also mapped into user-specific query and key spaces to compute the relation between each item pair. 𝐖_u^Q and 𝐖_u^K denote two user-specific parameter matrices for this space transition.













Considering the user number, adopting independent parameter matrices for each user will result in the issue of model parameter overload. Hence, we alternatively leverage user information to fine-tune a series of foundational parameters. Taking 𝐖_u^V as an example, its acquisition is as follows:


    𝐖_u^V = MLP(𝐡_u)·𝐔^V,

where 𝐡_u is the personal representation of user u learned in Graph RQ-VAE. It is projected through MLP into a scalar, measuring the bias introduced by user u on the foundational matrix parameter 𝐔^V∈ℝ^D× D. This strategy allows regulating the number of model parameters while maintaining user-specificity. 𝐖_u^Q and 𝐖_u^K can be obtained in the same way.

Notably, the relation-aware self-attention can be efficiently computed using the scaled dot product: 

    𝐗 = Softmax(𝐄(𝐖^Q𝐖^K^⊤ + 𝐖_u^Q𝐖_u^K^⊤)𝐄^⊤/√(D))𝐄(𝐖^V+𝐖_u^V).


The parameter matrices 𝐖^Q, 𝐖^K, 𝐖^V∈ℝ^D× D are unique to each attention head, contributing sufficient expressive capacity. While the user-specific parameters 𝐖_u^Q, 𝐖_u^K, 𝐖_u^V∈ℝ^D× D can be shared across all attention heads. 
To form the sublayer output, the outcomes from each head are concatenated, and a linear transformation is applied.

Overall, we use the relation-aware self-attention sublayer followed by a feed-forward sublayer as the Transformer encoder layer. Residual connections surround each sublayer, followed by layer normalization. As for the Transformer decoder, we maintain the default decoder layers and positional encoding, as the Transformer's output sequence is inherently ordered.





 §.§ Training & Inference


During model training, each observed interaction (u, i) forms a training sample. We input the atomic ID-based interaction sequence of user u into the Transformer encoder. To ensure generative recommendation, we deliberately exclude the ground-truth item i from the input sequence. Model parameters are learned by optimizing cross-entropy loss between Transformer decoder output and item i's Rec-ID. This loss function is commonly used for Transformer, and the indices of Graph RQ-VAE codebooks serve as vocabulary.

In inference with a well-trained Transformer, results are output step by step, where each step's conditional probability distribution depends on the already generated results. 
To maximize the product of conditional probabilities for the entire output, we incorporate beam search <cit.> to preserve K sequences with the currently highest probabilities at each time step. The beam number is configured to K for top-K recommendation. Through M steps of autoregressive decoding, beam search can ultimately generate K complete Rec-IDs. To prevent generating invalid Rec-IDs, we use constrained beam search <cit.> to limit the current token range based on prefixes.



 §.§ Discussion

Moreover, we discuss the inference time complexity of MMGRec compared with the most efficient method of traditional paradigm, i.e., Matrix Factorization (MF) <cit.>. Assuming the existence of I items, each represented by D-dimensional embeddings, MF requires a time complexity of 𝒪(ID) to complete all inner product computations and 𝒪(IlogK) to identify the top-K recommendation for a single user. In contrast, MMGRec utilizes autoregressive decoding and beam search for inference. A unit decoding process with H-layer MMGRec requires 𝒪(H(M^2D+MD^2)) in time complexity, where M represents the length of the output sequence (Rec-ID). Consequently, performing autoregressive decoding M times with K sequences incurs a time complexity of 𝒪(KH(M^3D+M^2D^2)). Additionally, a beam search with M steps introduces a time complexity of 𝒪(MKLlogK), with L denoting the vocabulary size. For ease of comparison, we can express the overall complexities as 𝒪(KH(M^3+M^2D)D) and 𝒪(MKLlogK). Upon closer examination, it suggests that the generative paradigm may reduce the inference time in case I≫ KH(M^3+M^2D)>MKL.



§ EXPERIMENTAL SETUP





 §.§ Datasets & Evaluation Metrics

To evaluate the effectiveness of our proposed model, we conduct extensive experiments on three benchmark datasets: MovieLens[https://movielens.org/.], TikTok[https://www.tiktok.com/.], and Kwai[https://www.kwai.com/.], which are widely used in multimodal recommendation study. The dataset statistics are summarized in Table <ref>.



  * MovieLens: This dataset was created for research by MMGCN authors. They collected movie trailers and descriptions based on the original dataset. 

Visual, acoustic, and textual features are extracted from frames, audio tracks, and descriptions using pre-trained models <cit.>, respectively.

  * TikTok: This dataset was released by the short-video sharing platform TikTok, including user-item interactions and multimodal features (visual, acoustic, textual) of items. According to the publisher's statement, the multimodal features are extracted from the raw data of short videos.

  * Kwai: The dataset was released by the short-video sharing platform Kwai, comprising user behavioral records and short videos' multimodal features (excluding acoustic features). Likewise, the multimodal features are obtained by pre-trained extractors.





For each dataset, we randomly select 80% of the historical interactions per user to constitute the training set, 10% for validation, and the remaining 10% for testing. The validation and testing sets are used for hyper-parameter tuning and performance evaluation, respectively. We adopt Recall (R@K) and Normalized Discounted Cumulative Gain (N@K) as evaluation metrics for top-K recommendation and preference ranking. Defaulting K=10, we report the average metric value for all users in the testing set.




 §.§ Baselines

To demonstrate the effectiveness, we compare our method with the following state-of-the-art baselines, briefly divided into CF-based (i.e., GraphSAGE, NGCF, GAT, and LightGCN) and multimodal (i.e., VBPR, MMGCN, GRCN, LATTICE, InvRL, and LightGT).


  * GraphSAGE <cit.> passes information along the graph structure and aggregates them to update each node's representation.

  * NGCF <cit.> encodes CF signals into representation learning by exploiting high-order connectivity from user-item interactions.

  * GAT <cit.> automatically learns weights for each node's neighbors and alleviates noisy information to improve GCN performance.

  * LightGCN <cit.> simplifies GCN components, utilizing a weighted sum aggregator as the graph convolution operation.

  * VBPR <cit.> incorporates multimodal information with MF framework to predict the interactions between users and items.

  * MMGCN <cit.> allocates an independent GCN for each modality, which learns modality-specific user preferences and item representations via the propagation of modality information.

  * GRCN <cit.> adaptively adjusts the interaction graph's structure according to the model training status, then applies graph convolution layers to distill informative signals on user preference.

  * LATTICE <cit.> discovers item relationships via multimodal features to learn a graph structure. Graph convolutions aggregate high-order affinities along the structure for recommendation.

  * InvRL <cit.> eliminates spurious correlations via heterogeneous environments to learn consistent invariant item representations across diverse settings, improving predictions of interactions.

  * LightGT <cit.> is a state-of-the-art Transformer-based model. It designs modal-specific embedding and layer-wise position encoding for effective feature distillation, learning superior user preference on item content for interaction prediction.



To ensure consistency, we adopt the publicly available implementations of the baselines. Note that we did not select the existing generative recommendation model TIGER <cit.> as baseline, since it only works with sequential recommendation rather than our task.



 §.§ Parameter Settings

We implement our MMGRec using Pytorch[https://pytorch.org/.] and Pytorch Geometric[https://pytorch-geometric.readthedocs.io/en/latest/.]. Model parameters are initialized with Xavier approach <cit.> and optimized using the SGD optimizer <cit.> with a batch size selected from {500, 1,000, 2,000, 3,000}. Hyper-parameters are tuned through grid search based on the results from the validation set. The optimal learning rate is searched from {0.0001, 0.0005, 0.001, 0.005} and ultimately set to 0.001. The L_2 normalization coefficient is searched within {10^-6, 10^-5, · · ·, 10^-1, 1}, with the optimal value set to 10^-5. Moreover, early stopping is adopted if Recall@10 on the validation set does not rise for 20 successive epochs. Without specification, user/item representations default to a size of  64 in all methods for fairness. As for the Graph RQ-VAE model, we adopt three-stage quantization to assign a three-tuple Rec-ID for each item. To avoid Rec-ID collision, we add a popularity token as the fourth. The size of each level codebook is tuned in {64, 128, 256}. When computing the loss ℒ_rqvae, we set β=0.25.




§ EXPERIMENTAL RESULTS

We conduct extensive quantitative and qualitative experiments to answer the following research questions:


  * RQ1: How does MMGRec perform on the multimodal recommendation task compared with state-of-the-art baselines?

  * RQ2: Is the technical choice of each component (i.e., item representations, Graph RQ-VAE, and relation-aware self-attention)  effective for MMGRec?

  * RQ3: How do hyper-parameter settings (e.g., depth of layer and number of heads) affect MMGRec?

  * RQ4: How does MMGRec compare with traditional paradigm methods regarding inference efficiency?




 §.§ Performance Comparison (RQ1)

Table <ref> presents the results of MMGRec and baselines over three experimental datasets. Besides, it reports the improvements and statistical significance test, which are calculated between our method and the strongest baseline (highlighted with underline). From the results, the main observations are as follows:



  * MMGRec consistently achieves the best performance in all cases. In particular, improvements over the strongest competitor w.r.t. NDCG@10 are 7.17%, 6.79%, and 6.58% in MovieLens, TikTok, and Kwai, respectively. Additionally, we conduct one-sample t-tests, which reveal that the improvements of MMGRec are statistically significant (p-value < 0.05). The primary difference between MMGRec and baselines lies in whether the recommendation paradigm is generative or traditional. Hence, we attribute the significant improvements to introducing a Transformer-based generative paradigm in multimodal recommendation.

  * MMGRec and the strongest baseline LightGT outperform other baselines by a large margin. This phenomenon validates that the Transformer is applicable to the multimodal recommendation. Nevertheless, LightGT focuses on improving representation learning by only utilizing the Transformer encoder. In comparison, our model exploits the generative capability of the complete Transformer, thus achieving superior performance.

  * Comparing the performance of MMGRec on two evaluation metrics, we find larger improvements in terms of NDCG. Namely, MMGRec's recommendation result is more accurate with respect to preference ranking. This illustrates that the generative paradigm is especially good at inferring the user's favorite items.









 §.§ Ablation Study (RQ2)







  §.§.§ Effect of Item Representation

To verify the necessity of item representations for MMGRec, we adopt two variants:


ID w/o: Without (w/o) item representation, the variant concatenates multimodal features as the input of RQ-VAE to assign Rec-ID.

Emb w/o: The variant replaces item representations in the Transformer encoder's input embedding with parameter embeddings.

Table <ref> displays the performance comparison between MMGRec and its variants, and we have the following observations:


  * The poor performance of ID w/o suggests that heterogeneous multimodal features cannot be directly used for Rec-ID assignment. The likely reason is the complexity of semantic relations among different modalities, necessitating prior integration through multimodal fusion. This aligns with the prior study about the importance of multimodal fusion in semantic comprehension <cit.>.

  * Despite parameterizing input embedding, Emb w/o underperforms and is inferior to utilizing well-trained item representations.  Hence, we infer that the Transformer struggles to learn suitable input embeddings directly from the generative recommendation task without pre-training. This is consistent with many Transformer-based NLP models that benefit from pre-trained word embedding like Word2Vec and Glove.






  §.§.§ Effect of Graph RQ-VAE

To study the importance of Graph RQ-VAE, we compare it with Hierarchical K-Means Clustering (HK-Means) <cit.> in Rec-ID assignment. HK-Means aims at establishing a cluster hierarchy, employing a top-down methodology that initiates with a single cluster including all items, then iteratively divides it into smaller clusters according to item representations. The hierarchical cluster indices of an item can be concatenated as the Rec-ID.
Experimentally, we adjust the clustering number to obtain optimal performance and comparable cardinality.




  * Table <ref> shows the performance comparison between employing Graph RQ-VAE and HK-Means. Graph RQ-VAE consistently outperforms HK-Means, illustrating the advantageous quantization ability. To explore the reason, we find that more ID collisions exist in HK-Means due to the clustering characteristic.

This illustrates that collisions negatively affect performance and should be handled reasonably. 

  * Therefore, we also compare our collision solution with the previous method (denoted by Random) that randomly extends one token to make Rec-ID unique. As the results recorded in Table <ref>, it is obvious that such a random method is consistently inferior to our solution. This verifies that the Transformer cannot accurately generate the last random token, while our method relatively circumvents this weakness.






  §.§.§ Effect of Relation-Aware Self-Attention

To investigate whether MMGRec can benefit from relation-aware self-attention, we consider two variants of our model:

Default PE: This variant preserves the default sinusoid-based positional encoding of Transformer. 

w/o PE: This variant entirely discards positional encoding. 

Note that our investigation of positional encoding is limited to the Transformer's encoder, while the decoder's requires no changes. Table <ref> exhibits MMGRec's performance with different position encoding approaches. We find that:


  * The variant without any positional encoding achieves poor performance. Even using absolute positions of input elements for positional encoding can improve the performance. This indicates the importance of positional information for the Transformer <cit.>. 

  * Compared to default positional encoding, relation-aware self-attention yields remarkable improvements, suggesting the effectiveness of leveraging pairwise item relations as positional information. 

This also validates the necessity of utilizing user information to model user-specific pairwise item relations, distinguishing our design from a standard self-attention head.








 §.§ Hyper-Parameter Analysis (RQ3)




To investigate the impact of Transformer layers on our model, we vary the layer number in the range of {1, 2, 3, 4} when fixing the self-attention head number to optimal. As a comparison, we test the LightGT with the same setting. Left of Figure <ref> plots the results w.r.t. Recall@10 on three datasets, we observe that: First, MMGRec outperforms LightGT as varying layer numbers across all datasets. This validates the stable superiority derived from the generative recommendation. Second, MMGRec can achieve quite good performance with fewer layers. This indicates that our generative model can be effective in a fundamental setting, while the embedding-based method needs stacking Transformer layers to guarantee performance.

Moreover, we investigate the impact of multi-head attention on our model by comparing 1-head, 2-head, 4-head, and 8-head self-attention blocks with LightGT. As shown in the right of Figure <ref>, we observe that MMGRec consistently outperforms LightGT. As the head number increases, the performance of MMGRec tends to stabilize after rising. In comparison, the performance of LightGT shows a continued downward trend. Theoretically, one head of self-attention models the relation among elements in a single aspect. Therefore, we infer that the generative model can handle more kinds of relations than the embedding-based method.







 §.§ Efficiency Study (RQ4)

Instead of computing and ranking similarity, MMGRec employs autoregressive decoding to directly generate Rec-IDs identifying items during inference. To assess the efficiency of the two inference approaches, we conduct comparative experiments using identical computing configurations and record their inference time. Figure <ref> presents an efficiency comparison between MMGRec and MF regarding average inference time per user across different-scale Kwai datasets. 1/8 Kwai denotes randomly selecting one in eight items from the complete dataset. The main observations are as follows:

In scenarios where the item count is low (e.g., 1/16∼ 1/4 Kwai), MMGRec exhibits inefficiency, primarily due to the basic computational demands of autoregressive decoding. However, as the number of items increases (e.g., 1/4∼ 1 Kwai), the inference time of MF experiences a significant rise. This escalation is attributed to the fact that item count linearly impacts the time complexity of inner product computation and sorting. In contrast, MMGRec demonstrates enhanced efficiency as its inference time remains relatively stable. This stability arises from the unchanged number of autoregressive decoding steps, even with an increase in the number of items. These findings suggest that the generative paradigm holds the potential to deliver more efficient inference for large-scale recommendation.





§ RELATED WORK



 §.§ Multimodal Recommendation

Research on multimodal recommendation aims to enhance recommender systems by leveraging multimodal information. Existing methodologies typically employ a combination of CF and content-based recommendation techniques <cit.>. Early studies aim to integrate multimodal content for more rational representation learning. For example, the pioneering work VBPR <cit.> maps visual information to the representation space and concatenates it with CF representations. 
Subsequent work like ACF <cit.> and UVCAN <cit.> adaptively select multimodal features using different levels of attention mechanisms to learn representations of users and items. In recent years, the research trend turns to exploring the relationship between modal information and users/items. For instance, the representative work MMGCN <cit.> introduces a modality-aware GCN into multimodal recommendation tasks to aggregate and propagate multimodal information on the user-item graph. HHFAN <cit.> utilizes a modality-aware heterogeneous information graph to explore more complex relationships among users, micro-videos, and multimodal information, thereby improving recommendation performance.

Despite their effectiveness, existing methods suffer from inherent limitations, such as overwhelming inference cost, insufficient interaction modeling, and false-negative issues, stemming from their embed-and-retrieve paradigm. To address these limitations, this study proposes a generative multimodal recommendation method.



 §.§ Transformer-Based Recommendation Model

Since its inception, Transformer has demonstrated outstanding performance across various domains, such as natural language processing <cit.> and computer vision <cit.>, yielding remarkable results. Owing to its proficiency in modeling contextual information, Transformer has emerged as a central focus in recommendation research. Researchers have developed Transformer-based recommendation models to capture item relationships and user behavior patterns in historical interactions <cit.>. PEAR <cit.> leverages the Transformer to capture interaction and contextual information from the history of clicked items. LightGT <cit.> lightens the Transformer and combines it with GCN to enhance user preference modeling by considering item multimodal correlations. Recently, Transformer has emerged as a significant contributor to the development of generative recommendation. DSI <cit.> signifies a pivotal transition in information retrieval towards a generative paradigm, being the first successful end-to-end Transformer application for retrieval tasks. Relying on a novel semantic ID representation, TIGER <cit.> implements a Transformer-based sequence-to-sequence model capable of directly predicting the semantic ID of the next item to achieve generative recommendation. 

Existing Transformer-based approaches are primarily limited to traditional representation learning or sequential generative recommendation. This study firstly introduces the Transformer with a generative paradigm to the field of multimodal recommendation.





§ CONCLUSION

In this work, we propose MMGRec, a novel Transformer-based model for multimodal recommendation. MMGRec explores the generative paradigm to address limitations inherent in the traditional paradigm, consisting of two essential components — Rec-ID assignment and Rec-ID generation. In Rec-ID assignment, we integrate item multimodal data with CF information and develop a Graph RQ-VAE to quantize them into Rec-ID for each item. In Rec-ID generation, a Transformer model is trained to directly predict item Rec-IDs for recommendation based on a user's interaction sequence. Experimental results demonstrate that MMGRec achieves state-of-the-art performance while maintaining conditionally efficient inference.

For future work, we plan to enhance MMGRec's performance by developing a more effective vector quantization method to mitigate Rec-ID assignment collisions. Additionally, we intend to explore the integration of large models <cit.>, such as ChatGPT, to improve user and item representations. Furthermore, we are interested in leveraging MMGRec's generative capability to tackle the cold-start problem in recommendation <cit.>.
















ACM-Reference-Format




































