Hi-Gen: Generative Retrieval For Large-Scale Personalized E-commerce Search
Yanjing Wu Yinfu Feng Jian Wang Wenji Zhou Yunan Ye Rong Xiao

	






















	



    Alibaba Group
    Hangzhou
    China

qinge.wyj@alibaba-inc.com



    Alibaba Group
    Hangzhou
    China
yinfu.fyf@alibaba-inc.com




    Alibaba Group
    Hangzhou
    China
eric.wj@alibaba-inc.com



    Alibaba Group
    Hangzhou
    China
eric.zwj@alibaba-inc.com



    Alibaba Group
    Hangzhou
    China
yunan.yyn@alibaba-inc.com



	Alibaba Group
	Hangzhou
	China

xiaorong.xr@taobao.com 































Leveraging generative retrieval (GR) techniques to enhance search systems is an emerging methodology that has shown promising results in recent years. In GR, a text-to-text model maps string queries directly to relevant document identifiers (docIDs), so it dramatically simplifies the whole retrieval process. However, when applying most GR models in large-scale E-commerce for personalized item search, we have to face two key problems in encoding and decoding. (1) Existing docID generation methods ignore the encoding of efficiency information, which is critical in E-commerce. (2) The positional information is important in decoding docIDs, while prior studies have not adequately discriminated the significance of positional information or well exploited the inherent interrelation among these positions. To overcome these problems, we introduce an efficient Hierarchical encoding-decoding Generative retrieval method (Hi-Gen) for large-scale personalized E-commerce search systems. Specifically, we first design a representation learning model along with metric learning to learn discriminative feature representations of items to capture both semantic relevance and efficiency information. Then, we propose a category-guided hierarchical clustering scheme that makes full use of the semantic and efficiency information of items to facilitate docID generation. Finally, we design a position-aware loss to discriminate the importance of positions and mine the inherent interrelation between different tokens at the same position. This loss boosts the performance of the language model used in the decoding stage. Besides, we propose two variants of Hi-Gen (Hi-Gen-I2I and Hi-Gen-Cluster) to support online real-time large-scale recall in the online serving process. Extensive experiments on both public and industry datasets demonstrate the effectiveness and efficiency of Hi-Gen. It gets 3.30% and 4.62% improvements over SOTA for Recall@1 on the public and industry datasets, respectively. Moreover, deploying Hi-Gen to a large-scale E-commerce platform leads to 6.89% and 1.42% improvements over the baseline model in the online AB-testing experiment in terms of recall and Gross Merchandise Value (GMV), respectively. In addition, Hi-Gen beats the basic Differentiable Search Index (DSI) model and BM25 in zero-shot learning scenarios, which proves its generalization capabilities.








[500]Recommender Systems
[300]Generative Retrieval
[100]Personalization
	
















Hi-Gen: Generative Retrieval For Large-Scale Personalized E-commerce Search
    Rong Xiao
    April 29, 2024
===========================================================================





§ INTRODUCTION


With the rapid development of online E-commerce in recent years, personalized information retrieval technology has become increasingly important. For example, a user who requests a query on E-commerce websites like Amazon and AliExpress[AliExpress is one of the largest B2C cross-border platforms under Alibaba Group.] can quickly retrieve a lot of relevant items from different seller stores. Information retrieval not only helps customers find their favorite items but also generates revenue for these business platforms. 


Existing retrieval techniques can be classified broadly into three categories: sparse retrieval <cit.>, dense retrieval <cit.>, and generative retrieval (GR) <cit.>. Sparse retrieval acquires documents by evaluating the relevance between query terms and document terms. Dense retrieval encodes queries and documents into high-dimensional vectors with a neural network and computes their similarity to retrieve relevant documents. In GR, the standardized pipelined retrieve-then-rank strategy for information retrieval is replaced by a simple sequence-to-sequence (Seq2Seq) strategy. The GR models in which generative Language Models (LMs) <cit.> play a key role allow for end-to-end training and can serve as a differentiable sub-component in complex neural models for building modern intelligent E-commerce systems, which can enhance the customer experience. In light of this, GR has gained widespread attention in both academia and industry.








The whole process of GR consists of a series of decisions, associated with the sub-problems of document representation, indexing, and retrieval. The performance of GR heavily relies on the choices and optimizations made for these sub-problems.


Firstly, in the pre-processing stage, several term-based and vector-based strategies have been proposed to represent a document. For term-based approaches, a few representative words or tokens are selected to describe the semantic content of a document. In contrast, vector-based approaches present documents in dense vectors and model their similarities with inner product or cosine similarity, which effectively solve the problem of similar semantic meanings but different word representations faced by term-based methods. In addition to the semantic information, other side information, such as item conversion efficiency information, is also significant in e-commerce scenarios. Nevertheless, there are relatively few works that explicitly learn both the semantic and efficiency information in document[In the field of E-commerce search, we treat a product as a special document, so we sometimes use the word document or item to express a product when there is no ambiguity in this paper.] representation. We call this problem the document representation problem.






Secondly, document representations are mapped into their corresponding docIDs in the indexing stage. The existing docID generation methods can be roughly classified into three groups: 1) unstructured atomic identifiers, 2) naively structured string identifiers, and 3) semantically structured identifiers. The most naive way to represent documents is to assign each document an arbitrary (and possibly random) unique integer identifier. We refer to these representations as unstructured atomic identifiers following the work <cit.>. Considering the limited expressive power of a single integer number, a more reasonable and clever way is to use a small group of arbitrary unique integers, words, or tokens to form docID. We refer to these as naively structured string identifiers. To capture some information about the semantics of its associated document, a hierarchical clustering process over document embeddings has been employed in  <cit.> to introduce a decimal tree, which guides the generation of semantically structured identifiers. Structural representation has a wide range of potential applications because structural relationships exist in many real-world applications. However, the current docID generation methods of most GR models for large-scale E-commerce searches ignore encoding crucial efficiency information about items. We refer to this problem as the docID generation problem.








Next, the retrieval process in GR is accomplished by decoding docIDs given an input query. When docIDs are formed as a series of organized tokens, decoding them is a structured prediction task where the position of each token is very important. A good structure prediction method should be able to capture dependencies between the output components and make joint predictions that respect these dependencies. It can lead to more accurate predictions compared to making independent predictions for each component. However, prior studies have not adequately discriminated the significance of positional information or well exploited the inherent interrelation among these positions <cit.>. In a recent related work <cit.>, Lian et al. began to consider the importance of different positions in docID by applying an additional hierarchical loss based on the document ID hierarchy at the decoding steps, but they ignored the inherent interrelation among different tokens at the same position. Here we use the term "docID decoding problem" to describe this issue.





Finally, directly using GR in online large-scale recall would encounter the service delay problem caused by the poor inference calculation performance. Most previous work in the field of GR only reported results on small datasets <cit.>. Thus, there is a lack of real-world application testing results on large-scale industry applications such as large-scale E-commerce search and recommendation systems. In this paper, we call this issue the large-scale recall problem.





To overcome the above-mentioned problems, we propose an efficient Hierarchical encoding-decoding Generative retrieval method (Hi-Gen) for large-scale personalized E-commerce search systems. The main contributions of our work are: 





 
        
  * We introduce a novel algorithm for docID generation in GR. Our method first designs a representation learning model along with metric learning to learn a discriminative feature representation of a document for maintaining both semantic relevance and efficiency information. Subsequently, the category-guided hierarchical clustering scheme is leveraged to make full use of the semantic and efficiency information of documents to facilitate docID generation.

        
  * The position-aware loss is proposed to enhance the performance of the language model used in the decoding stage. Our loss mechanism aims to discriminate the importance of positions and mine the semantic and efficiency differences among various tokens at the same position.
        
        
  * We propose two variants of Hi-Gen (Hi-Gen-I2I and Hi-Gen-Cluster) to support online real-time large-scale recall in the online service process. Hi-Gen-I2I rapidly expands generative retrieval results with item-to-item (I2I) recall algorithms, which are commonly used in modern search and recommendation systems. In contrast, the Hi-Gen-Cluster method exploits the hierarchical structure information encoded in docID by truncating the decoded result early in decoding to bring in large-scale recall.
        
        
  * In comparison to the original DSI, our methods achieve state-of-the-art performance across both public and industrial large-scale datasets. Furthermore, the practical usefulness of our methods in real-world search systems is evidenced by the improvements seen in online AB experiment metrics on a large-scale E-commerce platform.


     



§ RELATED WORK






Following the publication of Transformer Memory as a Differentiable Search Index(DSI) <cit.>, a notable number of GR approaches have been proposed <cit.>. The performance enhancement of GR can be attributed to three primary factors: firstly, through the refinement of the document representation; secondly, by enriching the docID encoding information; and lastly, by optimizing the employed language model. 


Document Representation. In <cit.>, a pre-trained content encoder was adopted to generate a semantic embedding for each item which has associated content features that capture useful semantic information. Lian  <cit.> utilized a transformer-based encoder-decoder network to jointly learn query embedding, document embedding, and personal embedding for learning the relationships among them. Wang  <cit.> introduced an additional query generation process to bind the information of document content through training the Seq2Seq model with generated queries and their corresponding document identifiers to make the identifiers aware of the document semantics. However, there are relatively few works that explicitly learn discriminative representation to encode both the semantic and efficiency information about the item.





DocID Generation. Generally, a docID can be expressed as a single token (or atomic integers) or as a series of tokens, each of which could be a semantic numeric string or any arbitrary string. Tay et al. <cit.> investigated three kinds of docID generation strategies (unstructured atomic identifiers, naively structured string identifiers, and semantically structured identifiers) and claimed that semantically structured identifiers lead to better indexing and retrieval capabilities for DSI. GenNet <cit.> learns to tokenize documents into semantic docIDs via a discrete auto-encoding approach. TIGER <cit.> introduces a hierarchical quantizer (RQ-VAE) that generates semantic ID representations for items. SE-DSI <cit.> takes the generated pseudo query with explicit semantic meaning, called Elaborative Description, as the docID. Nevertheless, these methods solely focus on the semantic information of the documents but ignore the efficiency information. To capture both the semantic and efficiency information of items, we develop a discriminative representation process and then employ category-guided hierarchical clustering to generate docIDs.

Language Model Optimization. To train the language model in GR, NCI <cit.> proposes a PAWA loss to tackle the significance variation problem when the same token appears in different places. PersonalTM <cit.> utilizes hierarchical loss to increase the impact of topper level clusters. Though they have acknowledged the specificity of the docID, the inherent semantic and efficiency information buried in different positions of the docID remains uncovered. Therefore, position-aware loss is suggested to distinguish the importance of different positions and discover the inherent interrelation between different tokens within the position. Since recent results suggest that some properties of large language models emerge only for very large model sizes, some existing work has tried to improve the LM by increasing model parameters or using better models <cit.>. 


















 


§ PROPOSED METHODOLOGY







	


 §.§ Overview

Figure <ref> depicts the overall architecture of Hi-Gen. Hi-Gen consists of offline and online stages. At the offline stage, a discriminative representation learning model is first employed to learn a fusion embedding for each item. Then the learned representation is encoded into a semantically and efficiently structured document identifier (docID) through a category-guided hierarchical clustering process. Subsequently, a pre-trained language model (LM) equipped with a position-aware loss is fine-tuned using the training data. The LM is fed with the user, query, and context-related features and learns to output the corresponding item docIDs. At the online stage, the fine-tuned LM model generates the most related top-K docIDs with a given query. Two variants of Hi-Gen (Hi-Gen-I2I and Hi-Gen-Cluster) are designed to rapidly expand results based on top-K decoded docIDs to support large-scale recall. In what follows, we provide detailed descriptions of the proposed Hi-Gen.





 §.§ Discriminative Representation Learning




Given a query, the E-commerce search system makes personalized product recommendations based on the user's historical shopping behaviors and preferences. In our work, we construct an E-commerce search algorithm model by taking into account four types of features: user, query, item, and context features. In particular, query features are extracted from both the original query and its corresponding bigrams. Item features consist of item ID, title, and some other statistical information. User features consist of user ID, tags, and some other user-related information. Context features include several user historical behavior sequences under a specified query, such as historical click and pay item sequences.

  
  

Using dense document representation to generate docID has been proven to be an effective method <cit.> in GR. However, the efficiency information about the item has never been taken into account in existing works. To eliminate this problem, we designed a discriminative representation learning model that captures both semantic relevance and efficiency information about the item. The model consists of two modules: 1) Atomic Embedding Learning Module, and 2) Metric Learning Module.







  §.§.§ Atomic Embedding Learning Module

With user, query, item, and context features as input, three atomic embeddings (semantic, common, and efficient) are produced in this module. Figure <ref> (a) illustrates the architecture of the atomic embedding learning module. For simplicity, we represent the output of the embedding layer for the user feature, query feature, and context feature in a batch as x_u ∈ℝ^B × d_u, x_q ∈ℝ^B × d_q × d_k, and x_c ∈ℝ^B × d_c × d_k, where d_u and d_k are embedding dimensions of each feature, and d_q and d_c are the sequence lengths for query and context, B represents the batch size in training and inference stages.


To capture the underlying factors that explain variations in item data, we explicitly group the original features of an item into three parts: 1) item semantic feature: the semantic information about an item, such as its title; 2) item ID: unique identifier for an item; 3) item efficient feature: some efficiency-related statistical indicators, such as click and pay count within a week. After embedding and feature dimension reduction processing, these three parts of features are converted into three low-dimensional atomic features: 𝐱_i_s∈ℝ^B× d_i_s, 𝐱_i_c∈ℝ^B × d_i_c and 𝐱_i_e∈ℝ^B× d_i_e, which are named as semantic embedding, common embedding, and efficient embedding, respectively. Notably, these three atomic item embeddings could have the same feature dimension (d_i_s=d_i_c=d_i_e). To capture both semantic and efficiency information of items, a multi-task two-tower embedding model <cit.> that consists of a user tower network and an item tower network is employed in which relevance prediction and click prediction tasks are deployed as downstream tasks as shown in Fig <ref>. 




In the user tower network, we utilize two attention modules to make the model learn and focus on the important information within user and query embeddings: UserSelfAttention and UserQueryAttention. In practice, we compute the attention function on a set of queries simultaneously, packed together into a matrix 𝐐. The keys and values are also packed together into matrices 𝐊 and 𝐕. Thus we computerize the matrix of outputs as:


    ℤ(Q, K, V) = reshape(softmax(Q·K^T/√(d_k) ) ·V)


Therefore, the UserSelfAttention and UserQueryAttention can be represented as 𝐙(𝐱_c, 𝐱_c, 𝐱_c) and 𝐙(𝐱_q, 𝐱_c, 𝐱_c). The dimensions of UserSelfAttention and UserQueryAttention are ℝ^B × (d_c × d_k) and ℝ^B × (d_q × d_k), respectively. The concatenation of the two attentions and x_u is fed into a multilayer perceptron (MLP). Thus, the hidden layer 𝐔 of the user tower network with dimension ℝ^B × d_e can be expressed as follows:

    U = MLP(concat(x_u, ℤ(x_c, x_c, x_c), ℤ(x_q, x_c, x_c))).



 




	


 
As for the item tower network, two MLPs are employed for relevance prediction and click prediction tasks. The common embedding of the item is shared between these two tasks. The outputs of these two tasks are formulated as follows:



    y_r    = 𝒩(𝐔) ⊙𝒩(MLP(concat(𝐱_i_s, 𝐱_i_c)))
    y_c    = 𝒩(𝐔) ⊙𝒩(MLP(concat(𝐱_i_e, 𝐱_i_c)))

where ⊙ is the dot multiplication operation and 𝒩(·) is the L2-norm normalization function.

To train the atomic embedding learning module, we optimize the loss function as follows:

    ℒ_1   = loss(y_r, y_r) + w_c × loss(y_c, y_c)

where y_r is the relevance label and y_c is the click label of the item. w_c is a hyperparameter that is used to balance the relevance task and the click task in training. loss(·) can be any suitable loss function. In this paper, we employ the cross-entropy loss function for Eq.<ref>.


















  §.§.§ Metric Learning Module
 Through the atomic embedding learning module, we acquire three atomic embeddings: 𝐱_i_s, 𝐱_i_c, and 𝐱_i_e.
However, since the two-tower network mainly focuses on modeling the similarity between item and user, the generated embeddings may be weak at distinguishing the semantic and efficiency information of similar items. As we know, metric learning can map the original features into some appropriate feature space where similar items are closer to each other, and dissimilar items are further apart <cit.>. Thus, it is a suitable choice to introduce metric learning to refine the learned atomic embedding features. 






Figure <ref> (b) illustrates the architecture of our metric learning module. We leverage an MLP to learn a compacted yet discriminative feature embedding from the combination of efficient, semantic, and common embeddings of the item as shown in Eq.<ref>. And, we denote the refined fusion embedding as I_a ∈ℝ^d, where d is the feature dimension.


    I_a = MLP(concat(𝐱_i_c, 𝐱_i_e, 𝐱_i_s))






Subsequently, a triplet margin loss is used to reduce the distance between items with the same user behaviors (click or pay) on the same exposure page view (PV) and increase the distance between items with different behaviors. We collect a large amount of online real-exposure page information to form a training data set PV = {PV_1, PV_2, …, PV_n} where PV_i = {(I_a^j, y_a^j) | j ∈ (1, 2, …, n_i)} and construct the following optimization function:



    ℒ_2 =    min ∑_i=0^n∑_𝐈_a^j∑_𝐈_a^j_n∑_𝐈_a^j_p
        max{m, dist(𝐈_a^j, 𝐈_a^j_n) - dist(𝐈_a^j, 𝐈_a^j_p)}
    
        s.t.   𝐈_a^j_n   = {𝐈_a^k∈ PV_i| y_a^k≠ y_a^j}  ∀ i, k, 𝐈_a^j
    𝐈_a^j_p   = {𝐈_a^k∈ PV_i| y_a^k = y_a^j}  ∀ i, k, 𝐈_a^j


where m is a positive hyper-parameter and dist(·) is a distance function. In this paper, we use the L2-norm distance as the distance function. Meanwhile, the Adam optimization algorithm is adopted to solve Eq.<ref>.




 §.§ Document Identifiers (docIDs) Generation


Items in a large-scale E-commerce search engine are naturally organized and stored according to a specific hierarchical category tree structure. For instance, a red long dress can be categorized as (clothing, dress) where clothing is the name of the upper category and dress is the name of the lower leaf category. Utilizing the hierarchical category tree aids in more efficient item retrieval because it brings in useful semantic information.


Using hierarchical clustering, the docID of an item can be acquired once the dense vector representation is given <cit.>. However, this method yields an unintelligible docID that is devoid of real category information. To solve this problem, we designed a category-guided hierarchical clustering method for document identifier generation, and thus the generated docID has a strong and clear semantic meaning.





As shown in Figure <ref>, we first classify all items according to the category tree. Items that belong to the same leaf category undergo hierarchical clustering. Each category has its distinct category ID. For example, dress can be assigned to a leaf category ID 202. In addition, we calculate an efficient score (click-through rate (CTR) score) for each item to calculate the average efficient score for each cluster in the efficiency layer. The results of an item encoded by the efficiency layer are appended to the end of its category path to produce the final docID. The proposed category-guided hierarchical clustering algorithm is detailed in Algorithm <ref>.


  

	
	


        



 §.§ Language Model With Position-aware Loss 



Our unique docIDs include three distinguishing characteristics: 1) Prediction errors on the first several digits of the docID will be more significant than those on subsequent digits; 2) Improved relevance results from choosing to predict a category at the semantic layer that is pertinent to the target category rather than one that is irrelevant; 3) Predicating a cluster with a similar efficient score to the target cluster at the efficiency layer will lessen the efficiency divergence. To address these issues, a position-aware loss is proposed.  
 

 
        
 

    



As shown in Eq.<ref>, the position weight in the t step (w_t^p) consists of the positional hierarchical, semantic, and efficient weights, which are represented as w_t^h, w_t^s, and w_t^e, respectively. Greater weight is given to the earlier token by w_t^h. w_t^s causes a greater loss at the semantic layer for the irrelevant category. At the efficiency layer, w_t^e attempts to generate tokens that are as efficient as the target token. λ_h, λ_s, and λ_e are hyperparameters that help to balance the effect of positional hierarchical, semantic, and efficient weight. The semantic layer's length is S, while the docID's length is L. The loss function for each token in the docID is represented by loss. dist(·) is a distance function that judges how similar clusters are in efficiency, while relevance determines whether two categories are semantically similar or not. E represents the efficient score which is calculated in Algorithm <ref>. In our work, dist(·) is defined as the difference of the Click-Through Rate (CTR), and bert-sentence-similarity[https://www.kaggle.com/code/eriknovak/pytorch-bert-sentence-similarity] is utilized as relevance. w_t^h is a decay function, we set it to e^L-t/∑_i=0^Le^i. In addition, we use the cross-entropy loss as loss(·).

    


    ℒ_3   = ∑_t=0^Lw_t^ploss(y_t | y_<t, x_u, x_q, x_h, y_t; θ) 
    
                s.t.   w_t^p    = λ_h ×w_t^h
                + λ_s ×w_t^s
                + λ_e ×w_t^e 
    w_t^s    =
            
            1,    relevance(y_t, y_t)=0 and t ≤ S
    
            0,     otherwise
            
    w_t^e    =  
            dist(E_y_t, E_y_t) ,    t > S
    
            0,     otherwise
            
    ∑_t=0^Lw_t^h   = 1, w_t^h > w_t+1^h, 𝐰_t^h > 0   ∀ t.







 §.§ Online Serving Optimization


In a realistic large-score personalized search engine, retrieval performance is influenced by both the quantity and quality of recalled items. However, the language model is unable to generate numerous docIDs due to high latency in inference. To solve this issue, we propose two variants of Hi-Gen (Hi-Gen-I2I and Hi-Gen-Cluster) for efficient online large-scale recall.


I2I retrieval <cit.> uses the consumer's recent behaviors as triggers to retrieve recommended items. Hi-Gen-I2I treats the limited items that are generated from Hi-Gen as triggers and uses an I2I retrieval method (Swing[Swing is an i2i algorithm proposed by Alibaba Group and widely used in the industry.]) to perform the recall process. Thus, it also can expand recall results quickly.


 
With category-guided hierarchical clustering, extended results can be produced efficiently and rapidly because items in the same cluster share similarities. All items in the same cluster are taken into consideration by the Hi-Gen-Cluster. Namely, Hi-Gen-Cluster-K regards items that share the same top K tokens in their docIDs produced by the language model as the expanded recalled items.

 



§ EXPERIMENTS SETUP

        
	
 


 


 §.§ Datasets
 
We conduct experiments on two datasets (AOL4PS <cit.> and AEDST[Our code and dataset will be publicly available.]) to evaluate Hi-Gen for large-scale personalized search. Below, we describe the details of these two datasets.




AOL4PS is an open-source large-scale dataset tailored for personalized search research, that contains the user, the query, the corresponding clicked documents (URLs), and timestamps indicating the timeline of each user's interaction history. Table <ref> illustrates the respective quantities in both the training and testing sets, detailing the number of samples, users, queries, and documents in each. To meet the input requirements of Hi-Gen, we extract the first two tokens of each URL (e.g., www.spiritplay.org represented as www.spiritplay) as a category and construct a one-layer hierarchical category tree. Furthermore, we observed a 65% overlap between the training and testing sets in this dataset. To assess the generalization capability of the Hi-Gen model, we excluded samples from the testing set that had already appeared in the training set for zero-shot experimental analysis. The specific approach aligns with the methodology employed by PersonalTM <cit.>.



 
AEDST is a billion-scale industrial dataset sourced from a real large-scale E-commerce search system, including the user, query, item, and click time information. Organized in chronological order of user behavior, we designate the samples from the first 7 days as the training set and the samples from the subsequent day as the testing set. The detailed distribution of quantities is shown in Table <ref>. A detailed analysis revealed that 87.33% of the samples in the testing set had never appeared in the training set. This phenomenon is attributed to the nature of large-scale E-commerce search systems, where users typically do not click or purchase the same item under the same query in a short period. Hence, the testing set of AEDST dataset can be directly utilized to evaluate the model's generalization performance.


        
	


 §.§ Baselines

	

        
        
        
        


We chose the classical retrieval model BM25[https://github.com/dorianbrown/rank bm25] and the classical generative retrieval model DSI <cit.> as baselines. Additionally, for the AOL4PS dataset, we included the state-of-the-art model PersonalTM <cit.> as a baseline, which incorporates personalized information. For the AEDST dataset, we also compared different-sized T5 <cit.> 
 (Small[https://huggingface.co/t5-small] and Base[https://huggingface.co/t5-base]) models. Subsequent sections will provide a detailed analysis of the experimental results.
        
 
	
	


 §.§ Implementation Details

 
 
 
 
 
 
All experiments are conducted on the Tesla A100 platform using the Adam optimizer. The representation learning model, metric learning model[https://github.com/KevinMusgrave/pytorch-metric-learning], and language model are assigned learning rates of 0.0001, 0.00001, and 0.00005, respectively. Correspondingly, their batch sizes are set at 512, 10, and 64. The representation and metric learning models yield embeddings with dimensions of 256 and 768, respectively. Fast-kmeans[https://pypi.org/project/fast-pytorch-kmeans/] in which k is 10 is utilized to produce the docID. In cases where the cluster size exceeds 100, the hierarchical clustering algorithm is executed recursively; otherwise, termination occurs. The embedding of dimension 768 is harnessed to generate the docID for all DSI baselines, originating from a pre-trained BERT <cit.>[https://huggingface.co/bert-base-uncased] model. For the proposed Hi-Gen, we utilized the following settings: (1) w_c: 1.0; (2) m: 0.1; (3) λ_h: 0.8; (4) λ_s: 0.1; (5) λ_e: 0.1; (6) S: 1. Predictions are obtained by decoding the language model using the beam search methodology. Additionally, all experiments adopt k-fold cross-validation, with k set to 3.
	


 §.§ Evaluation Metrics

 
 

The retrieval accuracy of the model for k items is denoted as Recall@k. This metric serves as a benchmark for evaluating the performance of various retrieval algorithms across diverse datasets. Additionally, six supplementary metrics are utilized to evaluate the effectiveness of generative retrieval in online testing: RecallNum, click-through rate (CTR), conversion rate (CVR), PayCount, gross merchandise value (GMV), and Latency. RecallNum signifies the total number of items retrieved by the online search system. CTR and CVR represent simplified forms of click-through rate and conversion rate, respectively. PayCount quantifies the number of transaction orders completed by users after selecting a sponsored result. The abbreviated form of gross merchandise value is referred to as GMV. Latency serves as a metric to quantify the processing time for retrieval.
	


§ EXPERIMENTAL RESULTS

        
	
	


 §.§ Main Results


Table <ref> illustrates the experimental outcomes for AOL4PS, providing compelling evidence that Hi-Gen surpasses other retrieval methods in large-scale personalized search systems. Hi-Gen beats the state-of-the-art model (PersonalTM) by 3.30% and 7.73% in Recall@1 and Recall@10, respectively. Especially, all generative retrievals including DSI outperform BM25 in Recall@1 and Recall@10.
	
	



We compare Hi-Gen with DSI and BM25 in AEDST, as indicated in Table <ref>. Hi-Gen outperforms the best baseline model (BM25) on Recall@1, Recall@50, and Recall@100, improving by 4.62%, 25.87%, and 28.37%, respectively. Compared to T5-Small, T5-Base has twice as many model transformer layers. Based on the research metrics of DSI and Hi-Gen, we could conclude that using T5-Base as the backbone outperforms T5-Small. It suggests that better results are obtained with more layers. Surprisingly, in AEDST, DSI performs even worse than BM25. The main reason is an abundance of zero-shot cases in AEDST. As seen in Table <ref>, this phenomenon also occurs in zero-shot circumstances of AOL4PS.
	


    


 §.§ Ablation Study
	
 
To figure out the roles of different components in the model, we performed ablation research on Hi-Gen in AOL4PS. We offer four Hi-Gen variants: 1) w/o position-aware loss, where the loss is equal to DSI; 2) w/o category-guided clustering, in which the clustering technique is the same as DSI; 3) w/o metric learning, where metric learning is replaced with ordinary concatenation; 4) w/o discriminative representation, in which the embedding is identical to DSI.

 
In Table <ref>, the removal of any one of the four components of Hi-Gen results in a drop in performance, indicating the effectiveness of each component. Based on the results of experiments, we discover that category-guided hierarchical clustering is the most important part of Hi-Gen. When category-guided hierarchical clustering is eliminated, Recall@1 and Recall@10 drop by 21.22% and 22.71%, respectively. Apart from category-guided hierarchical clustering, Hi-Gen also heavily relies on discriminative representation. Removing discriminative representation will result in a 17.00% and 18.30% decrease in Recall@1 and Recall@10, respectively. Furthermore, removing metric learning will decrease 8.02% and 8.86% in Recall@1 and Recall@10 in Hi-Gen. It proves that using discriminative representations learned through representation and metric learning, rather than BERT embedding, leads to significantly superior results for Hi-Gen. What's more, Hi-Gen will reduce Recall@1 and Recall@10 by 6.51% and 7.16%, respectively, when position-aware loss is not utilized.
	
	
	
	
	


 §.§ Zero-Shot Analysis

 
AEDST is a very sparse data set because zero-shot samples make up about 87.33% of the test data samples. The results of the experiment are shown in Table <ref>, where DSI performs far worse than Hi-Gen or even BM25. This result is in line with the findings published in  <cit.>. We conducted experiments on the AOL4PS zero-shot scenarios to better illustrate Hi-Gen's effectiveness in such a scenario. Amazingly, Hi-Gen outperforms all baselines on any metric, as Table <ref> illustrates. Moreover, compared to DSI, Hi-Gen yields notable gains in Recall@1 and Recall@10 of 36.78% and 41.07%, respectively.
	



 



 §.§ The Impact Of Hi-Gen Variants




 
    In the online serving phase, we propose two variants of Hi-Gen: Hi-Gen-Cluster and Hi-Gen-I2I. All products that have the same top K tokens as the 100 docIDs generated by the language model are designated as recalled items by Hi-Gen-Cluster-K. Figure <ref> illustrates how Recall@5000 and RecallNum of the Hi-Gen-Cluster increase as K decreases. Though Hi-Gen-Cluster-1 has improved recall@5000 by 36.13% compared to Hi-Gen, it recalls too many items, which is unbearable in the online system. Compared to Hi-Gen-Cluster-1, combining Hi-Gen-Cluster-2 and Hi-Gen-I2I (Hi-Gen-Cluster-2-I2I) is better since its RecallNum is reduced by 61.35%, but Recall@5000 is only reduced by 4.12%. Hi-Gen-Cluster-2-I2I has improved Recall@5000 by 0.3%, 4.55%, and 32.01%, respectively, in contrast with Hi-Gen-Cluster-2, Hi-Gen-I2I, and Hi-Gen.

    


 §.§ Online Experiments


	To examine Hi-Gen's performance in a realistic E-commerce search system, we implemented it in a real-word large-scale online E-commerce search engine. To ensure accuracy, Hi-Gen-Cluster-2-I2I is utilized to produce a sufficient number of recall items. A live experiment is conducted for a week using an A/B testing framework. In the control group, 4.0% of users are chosen at random and shown search results that don't come from Hi-Gen. An extra 4.0% users are chosen at random in the experiment group, and shown search results contain items produced by Hi-Gen. As indicated in Table <ref>, Hi-Gen increased RecallNum, CTR, CVR, PayCount, and GMV by 6.89%, 0.83%, 1.51%, 0.92%, and 1.42%, respectively. It implies that Hi-Gen has significant advantages in terms of retrieving efficient items in a practical large-scale E-commerce search engine. Furthermore, Hi-GEN can meet online needs without increasing latency when compared to current complex online retrievals because it just generates a limited number of items by language model.
	
	
	
 
	
	

§ CONCLUSION

 
	In this work, we propose Hi-Gen, a generative retrieval for large-scale personalized E-commerce search systems. Unlike other generative retrievals, Hi-Gen encodes efficient and semantic information during docID generation meanwhile concurrently making full use of positional information during the decoding stage. In the docIDs generation phase, a representation learning model along with metric learning is designed to learn discriminative feature representations of products to capture both
semantic relevance and efficiency information. Then, category-guided hierarchical clustering is proposed to enhance the semantics of the docID. Position-aware loss is suggested to imitate positional information during the decoding stage. Based on the experiment's results, Hi-Gen is capable of achieving state-of-the-art performance in open-source and industrial large-scale personalized search datasets. Furthermore, we utilize Hi-Gen in a practical, extensive E-commerce search system and demonstrate its efficacy via online A/B testing. Notably, Hi-Gen performs admirably at resolving zero-shot scenarios.
	
	
	
        ACM-Reference-Format
	
	

 

 
 
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
 
     
	
	
	

§ SUPPLEMENT EXPERIMENTS

 

 §.§ Add Fake Data

 
 We trying to add more fake data in the training data. From the experiments, we know that the metrics not improve not matter in all test dataset or zero-shot cases in AOL4PS.
 	

    

 
 

 §.§ using feature predict data

 
 we trying to test the effectiveness of the model in predicting the feature ground truth. 
 
	



        

 

 §.§ Model Parameter

 
 

 

 §.§ Inference Latency

 
  
 
  

 §.§ The Impact of Multitask For Embedding Learning



        

  

 §.§ The Impact Of Output Level

  
        




 §.§ The Impact Of Output Embedding

 
 
 
 

 §.§ The Impact Of Generative I2I


     
 
  

 
	


 
 

 
