Retrieval and Distill: A Temporal Data Shift-Free Paradigm for Online Recommendation System
Lei Zheng Ning Li Weinan Zhang Yong Yu








zhenglei@roro-ai.com


  RORO-AI PTE. LTD.
 Singapore
 Singapore



lining01@sjtu.edu.cn

  Shanghai Jiao Tong University
 Shanghai
 China



wnzhang@sjtu.edu.cn

  Shanghai Jiao Tong University
 Shanghai
 China

The corresponding author is Weinan Zhang



yyu@apex.sjtu.edu.cn

  Shanghai Jiao Tong University
 Shanghai
 China





Current recommendation systems are significantly affected by a serious issue of temporal data shift, which is the inconsistency between the distribution of historical data and that of online data.
Most existing models focus on utilizing updated data, overlooking the transferable, temporal data shift-free information that can be learned from shifting data.
We propose the Temporal Invariance of Association theorem, which suggests that given a fixed search space, the relationship between the data and the data in the search space keeps invariant over time.
Leveraging this principle, we designed a retrieval-based recommendation system framework that can train a data shift-free relevance network using shifting data, significantly enhancing the predictive performance of the original model in the recommendation system.
However, retrieval-based recommendation models face substantial inference time costs when deployed online.
To address this, we further designed a distill framework that can distill information from the relevance network into a parameterized module using shifting data.
The distilled model can be deployed online alongside the original model, with only a minimal increase in inference time.
Extensive experiments on multiple real datasets demonstrate that our framework significantly improves the performance of the original model by utilizing shifting data. 







































printacmref=false






Retrieval and Distill: A Temporal Data Shift-Free Paradigm for Online Recommendation System
    Yong Yu
    April 29, 2024
===========================================================================================





§ INRODUCTION



Click-through rate (CTR) prediction models play a significant role in modern recommendation systems. To enhance the accuracy of CTR models, deep learning-based approaches have been extensively employed <cit.>. However, the dynamic environment faced by the recommendation systems, with the data distribution evolving over time, poses significant challenges to the effectiveness of these models <cit.>.

To tackle this challenge, prior research has undertaken investigations from two distinct perspectives, namely incremental learning and behavior sequence modeling. 
On one hand, incremental learning techniques delve into updating the models that make these predictions as new data comes in, without having to start from scratch every time <cit.>. 
This means the models learn and improve gradually as they get access to more recent data. 
However, the process of learning neural network parameters depends on iterative updates based on gradients, which slowly infuse supervision data into the model weights using a minimal learning rate. 
This poses a significant challenge for large parametric recommendation models in swiftly adapting to shifts in distribution.
On the other hand, behavior sequence modeling focuses on understanding how users' behaviors change over time by looking at the sequence of actions they take. 
In this area, various network structures have been proposed, including recurrent
neural networks (RNNs) <cit.>, convolutional neural networks (CNNs) <cit.>, and memory networks <cit.>.
Moreover, the attention mechanism has emerged as the predominant approach for modeling item dependencies, with several notable contributions such as SASRec <cit.>, DIN <cit.>, DIEN <cit.>, and BERT4Rec <cit.>.
These studies aim to identify sequential patterns in user behavior, thereby facilitating the prediction of future preferences based on historical activities. However, these studies fail to directly tackle the issue of distribution shift during training time, and these high-capacity models significantly increase the online workload and inference burden.








All of the above-mentioned models either update the model using the latest data <cit.> or use past shifting data as a reference for retrieval <cit.>, without directly using the shifting data for training model parameters.
However, directly using the temporally shifting data for model training is challenging, as an example illustrated in Figure <ref> based on a public dataset of Tmall, an international e-commerce platform.

In this dataset, data spanning Nov. 11-12, 2015 is employed as the test set.
The x-axis represents the training set from the indicated date to Nov. 10, 2015. We train a common deep CTR model PNN <cit.> using different training sets to make predictions. The y-axis indicates the AUC performance of the model on the test set.
We can observe that utilizing data over Nov. 1-10, 2015 for predictions on the test set yields the best results, whereas adding data from earlier time periods will negatively impact the results due to data shift.

Therefore, it is crucial to find an effective method that can directly utilize these data for model training.

Inspired by the recent proposed retrieval-based tabular data prediction models <cit.>, we found that similar data from historical records can significantly enhance the predictive capability and generalizability.
In other words, although the probability distribution modeled by the CTR is time-variant, the probability distribution of the association between current user behavior data and similar historical data can remain invariant over time if the search space is fixed.

If we design a neural network to model the probability distribution of data similar to the current query, this neural network is temporal data shift-free. Thus, the shifting data can be used to train such a neural network.
By incorporating this neural network with the original CTR model, we can obtain an aggregated prediction model that has been trained with shifting data.

In this paper, we propose the Retrieval and Distill paradigm (RAD) to address the challenge of ineffective training of a CTR model over shifting data.
The RAD paradigm generally consists of two major components, namely the Retrieval Framework and the Distill Framework. 
We first establish a Retrieval Framework to assist the original model in making predictions using similar data within the shifting data. 
We refer to the neural network in Retrieval Framework that retrieves and outputs the logits of similar data as the relevance network. 
By pre-training this relevance network using shifting data and then merging the original CTR model and relevance network, we can significantly enhance the predictive capability of the original CTR model. 
However, the added relevance network in the Retrieval Framework, due to its retrieval process (such as BM25 <cit.>), is difficult to deploy online due to its inference time, similar to other retrieval-based CTR models <cit.>. 
To address this difficulty, we further build the Distill Framework to distill knowledge from the relevance network using shifting data, obtaining a parameterized student network called search-distill module. 
By aggregating the original CTR model and the search-distill module, we finetune the Distill Framework with unshifting data, e.g., the latest training data.
This yields the final model that can be deployed online for light workload and efficient inference.



In sum, the technical contributions of this paper are threefold.

    
    
  * We highlight the significant impact of the data shift phenomenon in recommendation systems, which is not addressable by simply increasing training data or the model capacity. To deal with this issue, we propose the Retrieval and Distill (RAD) paradigm, where the recommendation model is enhanced by incorporating shifting data into the model. To the best of our knowledge, RAD is the first paradigm that successfully utilizes shifting data for model training and lightweight deployment.
    
  * As the rationale of RAD, we raise the theorem of Temporal Invariance of Association, which provides effective guidance on how to utilize shifting data.
    
  * We design an innovative distillation approach that distills information from the retrieval framework of RAD. This approach optimizes both the time complexity and space complexity of the retrieval process to O(1), which largely pushes forward the practical usage of retrieval-based models for online recommendation systems. 
    






Extensive experiments on multiple real datasets demonstrate that RAD significantly improves the performance of the original model via effective training over shifting data.
RAD offers a new paradigm in the field of recommendation systems, enabling models to train using both shifting and unshifting data, opening up a new pathway for enhancing the performance of CTR models by increasing the amount of training data.








§ PRELIMINARIES


In this section, we formulate the problem and introduce the notations. 
In the context of recommendation systems, we represent the dataset as a tabular structure, denoted by 𝒟, comprising F feature columns 𝒞 = {C_i}_i=1^F alongside a singular label column 𝒴. 
The feature columns encapsulate attributes such as "Age," "City," and "Category."
Each individual entry within table 𝒟 corresponds to a sample s_z. 
A sample is characterized by an amalgamation of multiple features along with a label, articulated as s_z = (x_z, y_z), where x_z = {c_i^z}_i=1^F. 
Accordingly, table 𝒟 is defined as an ensemble of N samples, expressed as 𝒟 = {s_z}_z=1^N. 


A click-through rate (CTR) model can be mathematically represented as a conditional probability, P(y|x), where y is the event of a click, and x represents the features. The goal of the model is to accurately estimate this probability given the features x, which can include information about the item, the user, and the context.


Temporal Invariance of Association.


In the paper, we model P(y|x) as P(y|x, x_s), where x_s represents documents similar to x within a fixed search space ℰ.
Given that the probability distribution of P(y|x, x_s) changes over time, while the probability distribution of P(x_s|x) remains constant, we can utilize the data come from the time-varying P(y|x, x_s) to train a model that represents P(x_s|x). 
This process allows us to capture the temporal dynamics in P(y|x) while leveraging the time-invariant properties of P(x_s|x), thereby enhancing the model's adaptability to temporal changes in the future.



We partition the dataset by date into three segments. 
The last few days, denoted as 𝒟_[t+1, t+Δ], serve as the test set, 𝒟_test. 
Data within a window of d days prior to the test dates is considered unshifting data, which is 𝒟_[t - d + 1, t] and is used as the training set, denoted by 𝒟_train. 
The remaining portion, 𝒟_(-∞, t - d], is designated as the search space, 𝒟_shifting.

The Retrieval and Distill (RAD) framework comprises two components. 
The first component, the Retrieval Framework, employs a retrieval-based method to assist any original model in addressing the data shift issue. 
However, retrieval-based models face significant efficiency challenges. 
Therefore, the second component, the Distill Framework, utilizes distillation to eliminate the retrieval process. 
We will introduce these two components separately in the following.

Retrieval Framework
The objective of the Retrieval Framework is to employ a relevance network, which incorporates non-parametric retrieval, to assist any original model F(x) in predicting the label. The formulation of the Retrieval Framework is as follows:

    ŷ_̂R̂ = f_θ(F(x), x, R(x)),

where R(x) is designed to fit P(x_s|x) and is referred to as the relevance network and f_θ which approximate the P(y|x, x_s) is data shift sensitive.

To obtain a model, R(x), that approximates the probability distribution P(x|x_s), we employ a Retrieval Framework, devoid of F(x), named the teacher network. The formulation of the teacher network is as follows:

    ŷ_̂T̂ = T_γ(x, R(x)).


According to Theorem <ref>, The R(x) component within T_γ, trained using shifting data, is invariant to data shift.
Therefore, to obtain an R(x) that has been trained with shifting data, T_γ can be minimized by the following loss function:

    ℒ_pretrain =  ∑_x_z ∈𝒟_shiftingCE(y^z, ŷ_T^z),

where CE is binary cross-entropy. This step obtains the model R(x) that approximates P(x_s|x).
 

Once we have obtained an R(x) that has been pretrained with shifting data, we can utilize R(x) in the training of the Retrieval Framework. We use unshifting data to minimize the following loss function:

    ℒ_retrieval =  ∑_x_z ∈𝒟_trainCE(y^z, ŷ_R^z),

where CE is binary cross-entropy. In this step, we obtain the Retrieval Framework, f_θ, which approximates P(y|x, x_s).

Distill Framework
Although the Retrieval Framework can leverage shifting data to significantly enhance the original model, the deployment of retrieval-based methods presents substantial challenges in terms of storage space and inference time in an online environment. Therefore, we have designed the Distill Framework to eliminate the non-parametric search process.

Distill Framework can be formulated as:

    ŷ_D = f'_ϕ(F(x), x, r(x)),

where r(x) is distilled from R(x) pretrained using loss function  (<ref>), which we refer to as the search-distill module. Similar to f_θ, f'_ϕ approximates P(y|x, x_s) and can also only be trained using unshifting data.


We assume

    w_R = R(x),

where w_R is the last layer logits of R(x) and assume

    w_r = r(x),

where w_s is the last layer logits of r(x).
The loss function used to distill knowledge from R(x) to r(x) can be written as:

    ℒ_KD =  ∑_x_z ∈𝒟_shiftingMSE(r(x), R(x))
    = ∑_x_z ∈𝒟_shiftingMSE(w_r^z, w_R^z),

where MSE is the Mean Squared Error loss function.



Once we obtain the search-distill module, we can use unshifting data to derive a final Distill Framework using the following loss function:

    ℒ_distill =  ∑_x_z ∈𝒟_trainCE(y^z, ŷ_D^z),

where CE is binary cross-entropy loss.









§ THE RAD PARADIGM


In this section, the details of RAD will be presented. We start with an overview of the entire paradigm's structure at Section <ref>, followed by detailed expositions of the Retrieval Framework and Distill Framework in Sections <ref> and <ref>, respectively.












 §.§ Overview

Traditional CTR models have not accounted for the data shift issue, and they cannot enhance performance by merely adding shifting data <cit.>. 
Training only with unshifting data neglects a significant amount of information.
Our proposed Retrieval and Distill paradigm aims to devise a universal solution that utilizes shifting data to enhance the performance of existing models. 
RAD is generally divided into two parts.
The first part incorporates a Retrieval Framework to introduce shifting data into the datastore during training. 
Concurrently, as per theorem <ref>, we pretrain the model's relevance network with shifting data and then transfer it for use. 
This approach substantially improves the model's performance without altering its original structure. 
However, including a non-parametric search component in the relevance network poses deployment challenges for online recommendation systems. 
Therefore, the second part is a Distill Framework tasked with transforming the relevance network into a parametric neural network model. 
We distill a simple neural network (search-distill module) using shifting data and the relevance network as the teacher network. 
Ultimately, we merge the search-distill module with the original model and finetune it using unshifting data to yield the final online deployable model. 
The following sections will elucidate these modules in detail.




 §.§ Retreival Framework
 
The Retrieval Framework aims to employ a relevance network that utilizes shifting data in conjunction with the original model for CTR prediction.
According to theorem <ref>, the relevance network is temporal data shift-free; therefore, as an initial step, we pretrain the relevance network using shifting data with a teacher network, which is a Retrieval Framework that does not contain the original model. In the second step, we jointly train the relevance network and original model using unshifting data.
Next, we will detail the neural network structure of the Retrieval Framework,  followed by its training method.



  §.§.§ Framework Model Structure

The neural network structure of the Retrieval Framework can be seen in Figure <ref>.
The Retrieval Framework aggregates the outputs of both the teacher network and the original model, where the teacher network specifically combines the original input with the relevance network. We shall now explicate the original model, relevance network, and teacher network.

Original Model We denote s_z as any tabular data entry within the dataset 𝒟, where the feature part of s_z is represented as x_z, and the label part as y_z, with x_z = {c_z^i}_i=1^F comprising F features. Traditional CTR models take such an x_z as input and output a probability.
We represent the original model as O, with the last layer logit denoted by w_o. 

In the traditional model, the probability prediction is given by 

    ŷ_̂Ô^̂ẑ = O(x_z) = sigmoid(w_o),

where w_o is the input of the aggregation layer of the Retrieval Framework.


Relevance Network
The relevance network features an embedding layer shared with input x_z, a search engine, and an attention-based aggregation layer.
Within our framework, we utilize shifting tabular data entries as the search space, with BM25 <cit.> serving as our search algorithm.

In the BM25 algorithm, we treat any sample x_z as the query and a key x_d in the datastore as the document. As such, the ranking score can be calculated as

    RankScore(x_z, x_d)    = ∑_j=1^M IDF(a_j^z)·1 (x_j^z = x_j^d) ,
    IDF(x_j^z)    = logN-N(a_j^z) + 0.5/N(a_j^z) + 0.5,

where 1(·) is the indicator function, N is the number of data samples in 𝒟_shifting, and N(a_j^z) is the number of data samples that have the categorical feature value x_j^z. By ranking the candidate samples using BM25, we obtain the
top K entries as the retrieved set 𝒮(x_z).


For each item within 𝒮(x_z), an attention mechanism is applied to calculate its aggregation parameter

    α_k = exp(x_k^⊤ W x_z)/∑_j=1^Kexp(x_j^⊤ W x_z),


The elements of 𝒮(x_z) are then consolidated into a  logit w_R : 

    w_R = ∑_k=1^Kα_k · x_k,    s_k ∈𝒮(x_z),

where w_s is the input of the aggregation layer of the teacher network.

Teacher Network The teacher network is a Retrieval Framework that does not include the original model. From Figure <ref>, we can observe that the teacher network forms the right portion of the diagram.
The teacher network serves as an intermediary bridge for training the relevance network.
The teacher network takes x_z and the relevance network output as inputs, producing a probability as output.
Drawing inspiration from RIM <cit.>, we incorporate a factorization machine module within the aggregation layer of the teacher network. 
This aggregation layer is designed to effectively merge the input x_z  with the aggregated w_R, thus improving the teacher network's capability to discover the hidden relationships between x_z and the returned S(x_z) from the search engine.




  §.§.§ Pretrain with Shifting Data

The relevance network models the probability distribution P(w_R|x), where w_R is the logit representation of x_s. As established by theorem <ref>, the probability distribution represented by the relevance network is time-invariant, which allows us to pretrain it using shifting data. 
To obtain the relevance network, we initialize a separate teacher network. The teacher network is trained using the dataset 𝒟_shifting and with loss function  (<ref>).
Although the separate teacher network trained in this manner performs poorly on the test set 𝒟_test due to data shift—often hovering around 0.5 in our experiments—the relevance network is transferable.





  §.§.§ Finetune with Unshifting Data


After obtaining the pretrained relevance network, we integrate it with the original model and the target query x_z to form the Retrieval Framework. This framework is then trained using 𝒟_train with loss function  (<ref>), essentially finetuning it. 



 §.§ Distill Framework

Although the Retrieval Framework alone allows us to significantly enhance the performance of various existing CTR models by utilizing shifting data, this improvement depends on the relevance network's inclusion of BM25 search, which can be computationally intensive due to the typically large volume of 𝒟_shifting data and the often linear time complexity of the search process.
This presents a substantial challenge for deploying the Retrieval Framework in an online environment.
To optimize the time complexity of the Retrieval Framework, we propose a Distill Framework designed to distill the relevance network and remove its search component. 
The resulting module is termed the search-distill module. 

Next, we start by delineating the structure of our model, followed by a description of the Distill Framework's training process.
Within the Retrieval Framework's pretraining step, we achieve a pretrained relevance network.
In the Distill Framework, this network undergoes distillation using 𝒟_shifting. Subsequently, the distilled search-distill module is combined with the original model through an integrated multilayer neural network for aggregation and subsequent finetuning using 𝒟_training.



  §.§.§ Framework Model Structure

The neural network structure of the Retrieval Framework can be seen in Figure <ref>.
The network architecture of the Distill Framework is similar to that of the Retrieval Framework. We replace the relevance network with the search-distill module. 

From Figure <ref>, we can observe the form of the teacher network in the Distill Framework.
The teacher network in the Distill Framework differs somewhat from that in the Retrieval Framework. We removed the cross layer from the teacher network in the Retrieval Framework to distill only the relevance network portion and added two neural network layers after the attention layer. 
We use this teacher network to obtain a relevance network for distillation.
 
The search-distill module is a multilayer neural network equipped with an embedding layer. Akin to the original model, the search-distill module takes the feature portion of tabular data x_z as its input. The output of the search-distill module is a logit of the same size as that of the final output from the relevance network.







  §.§.§ Pretrain and Distill with Shifting Data

From theorem <ref>, we understand that the relevance network operates as a temporal data shift-free module.
Like the relevance network, the search-distill module is also used to approximate the probability P(x_s|x), which can be trained using shifting data.
We first obtain the relevance network using the teacher network and loss function (<ref>).
Then, we employ 𝒟_shifting and loss function (<ref>) to perform distillation on the final logit layer of the search-distill module. The distillation process can be observed in Figure <ref>. 
We employ the mean squared error loss function to align the final logit layer of the search-distill module with that of the relevance network.




  §.§.§ Finetune with Unshifting Data

After obtaining a distilled search-distill module, we integrate the original model, the input x_z, and the search-distill module using a multilayer neural network for aggregation. 
This network is then trained with unshifting data and loss function (<ref>), allowing us to finetune and enhance the original model with insights gleaned from shifting data without significantly increasing the deployment overhead of the original model.






§ EXPERIMENTS

In this section, we evaluate the RAD paradigm across multiple datasets using various traditional CTR models. 
We measure the efficacy of the RAD paradigm using the conventional CTR prediction task. We select two sets of classic CTR models: one representing factorization models and the other representing sequence prediction models. Each set of models is tested on three real-world datasets.

This section starts with five research questions (RQs) guiding the subsequent discourse.




  RQ1 Does the RAD paradigm enhance the performance of existing CTR prediction models?

  RQ2 Can distillation functions be applied to the distillation of non-parametric models?

  RQ3 Can the relevance network, trained using shifting data, be transferred to predict future data?

  RQ4 How does the RAD paradigm affect the run time of the original model?

  RQ5 How do different distillation loss functions affect the RAD paradigm?






 §.§ Datasets

In order to evaluate the performance of RAD, we conduct extensive experiments for CTR prediction tasks on three real-world large-scale recommendation datasets from Alibaba and Ant Group, i.e., Taobao, Tmall and Alipay[https://tianchi.aliyun.com/dataset/x, where `x' is `649', `42', and `53' for Taobao, Tmall, and Alipay, respectively.]. These datasets consist of sequential user-item interactions in e-commerce scenes. 
Table  <ref> is the information for our split dataset.







 §.§ Evaluation Metrics

In the CTR prediction task, the area under the ROC curve (AUC) and negative log-likelihood (LogLoss) are widely used metrics. AUC measures the probability that a randomly chosen positive sample will be ranked higher than a randomly chosen negative sample, while Logloss quantifies the difference between the predicted probability and the actual label. We choose these two metrics to evaluate the model performance. 



 §.§ Compared Methods

To evaluate the effectiveness of our proposed architecture, we choose six commonly used and strong models as backbones, which can be grouped into two classes. 



  §.§.§ Feature interaction based Models

The first class includes some traditional feature interaction based tabular models. DNN represents multilayer perceptron (MLP), a simple feedforward artificial neural network. DeepFM <cit.> is an architecture that combines the factorization machine and deep neural network. PNN <cit.> is a product-based neural network which captures both inter-field interactive patterns and high-order feature interactions. 



  §.§.§ Sequential based Models

The second class contains several sequential end-to-end deep models. Transformer <cit.> is a sequence-to-sequence model based on self-attention mechanism. Here, we use its encoder to model users’ history behaviors. DIN <cit.> and DIEN <cit.>are two attention-based sequential models, while DIEN adds a recurrent neural network based on DIN.












 §.§ Overall Performance (RQ1)

The results are listed in table <ref>. The results reveal the following insights.
i) In our experiments, sequence models, except the original Transformer model, outperformed factorization models. This indicates that increasing data volume is beneficial for recommendation systems.
ii) The Retrieval Framework utilizes shifting data as a search space and pretrains the relevance network with shifting data, significantly enhancing performance beyond the original model. This demonstrates the substantial benefit of shifting data for the original model.
iii) The Distill Framework employs the distilled relevance network to assist the original model. While predictive performance surpasses the original model, it falls short of models augmented by the Retrieval Framework, suggesting that the distilled relevance network retains valuable information, albeit with less incremental information than the retrieval model.
iv) The performance of feature interaction-based models within the Distill Framework has already surpassed that of the original sequence models. Moreover, as illustrated in Figure <ref>, the distillation method can enhance model performance without significantly increasing the inference burden



 §.§ Applying Knowledge Distillation to Non-Parametric Models (RQ2)

Knowledge distillation is traditionally employed for transferring knowledge between parametric models, typically from a complex, larger model to a simpler, smaller one. 
In this context, we extend the application of knowledge distillation to transfer knowledge from a non-parametric model (BM25 search) to a simple neural network. 
As illustrated in Table <ref>, distilling information from the retrieval-based relevance network into the search-distill module demonstrates that the distilled student model can still significantly enhance the performance of the original model. However, compared to the direct use of the non-parametric model, performance is considerably reduced.
There is significant room for improvement in distilling information from non-parametric models into parametric models.





 §.§ Temporal Invariance of Association (RQ3)


Table <ref> showcases a group of experiments designed to validate the theorem  <ref> of Temporal Invariance of Association. 
The relevance network aids the original model in prediction in two ways: the additional information provided by the retrieved similar data x_s, and the information gain brought by pretraining with shifting data. 
To eliminate the influence of x_s on the results during testing, we added D_train pretraining for comparison. 
Similarly, to rule out the impact of finetuning on the relevance network, we fixed the parameters of the relevance network during the finetuning phase.
By comparing the data in the table, we find that: i) The relevance network pretrained with shifting data performs better than the model pretrained only with train data. 
This indicates that the relevance network trained with shifting data indeed learns time-invariant information in the time-varying data distribution. 
Theorem <ref> is validated. 
ii) The relevance network trained with 𝒟_shifting performs better because whether using 𝒟_shifting or 𝒟_train, both are fitting P(x_s|x), and shifting data has a much larger volume. Hence, the trained relevance network performs better. 
Thus, we obtain a network that enhances model training effectiveness by increasing the volume of data, unlike the network in figure <ref>, where increasing the volume of data would decrease model performance.















 §.§ Performance Efficiency Study (RQ4)


Recently, a range of retrieval-based models have been introduced in the field of recommender systems, some of which utilize discrete features for retrieval, such as SIM <cit.> and RIM<cit.>, while others employ continuous features, like DERT<cit.>. Due to the significant time overhead associated with retrieval, these models face challenges in being directly deployed online. In Figure <ref>, we compare three methods for augmenting relevant information. As RAD employs distillation to bypass the retrieval process, it incurs no retrieval time. On the other hand, directly aggregating the search-distill module with the original model increases the inference burden on the original model. However, as shown in Figure <ref>, the additional inference load introduced by incorporating the search-distill module is not significant, and the time overhead it introduces is considerably less than the burden added by using sequence models for model inference.






 §.§ The Study of Distill Loss Functions (RQ5)

In Table <ref>, we compare the impact of various distillation functions on the Distill Framework.
For simplicity, our experiments were conducted using only three models based on feature interaction. 
Our findings indicate that the performance enhancement provided by different distillation loss functions is broadly similar, suggesting that the quantity of information extracted primarily depends on the shifting data itself. 
However, slight variations exist in the performance of different loss functions across datasets. The  Mean Squared Error Loss Function generally performs slightly better on Tmall, the Kullback-Leibler Divergence Loss Function shows slight superiority on Alipay, and the cosine similarity loss function performs marginally better on Taobao. When addressing different data distributions, selecting an appropriate distillation function tailored to the specific dataset can further enhance the distillation effect.
















































§ RELATED WORK

Click-through rate (CTR) prediction
is crucial in the fields of online advertising, recommendation systems, and information retrieval.
Many deep learning models for CTR prediction primarily concentrate on investigating the feature interactions within a single instance <cit.>. 
The Wide&Deep network <cit.> constructs fully connected (FC) layers based on field feature embeddings to extract feature interactions implicitly. 
CCPM <cit.> employs convolution operators to probe local feature interaction patterns from a list of feature embeddings. 
PNN <cit.> suggests executing inner or outer product operations on each pair of field feature embeddings, followed by FC layers for further implicit feature interaction extraction. 
DeepFM <cit.> evolves from the factorization machine (FM) <cit.>.

Recently, graph neural networks have also shown great potential in recommendation scenes by constructing data into graph structures and performing message passing <cit.>.
Despite the numerous models proposed in this direction, the model capacity might be constrained as the model's input is a single instance, necessitating the model to encode all knowledge into the parameters. 
Our work employs these deep models as the base model of the framework.




Retrieval-based methods for CTR prediction
are typically derived from sequential recommendation tasks.
where it's common to use a recurrent neural network (RNN) to construct user profile representations for recommendation predictions. 
However, when the user's historical behavior sequence becomes significantly long, RNN architectures struggle to memorize and utilize early behavior patterns <cit.>. 
Therefore, retrieval methods serve as a practical alternative to lengthy recurrence and attention operations. 
UBR <cit.> creates a search index from the user's historical sequence data and employs reinforcement learning to generate a search query based on current prediction data. The retrieved data is then aggregated through an attention operation and input into an MLP for label prediction. 
SIM <cit.> executes both hard search based on item category information and soft search based on item embedding vectors, using the DIEN model <cit.> as the backbone. 
RIM <cit.> expands the research scenario from sequential recommendation to generic tabular data prediction. The query is the target row data, and the retrieved data instances are the relevant rows in the table, with BM25 <cit.> serving as the predefined relevance function between two rows. DERT <cit.> is the pioneering work to incorporate dense retrieval into recommendation systems. Retrieval-enhanced machine learning is now considered a significant frontier in machine learning research <cit.>.


Knowledge Distillation (KD)
<cit.> seeks to transfer the 'dark knowledge' from a teacher model to a student model and is extensively used in model compression and knowledge transfer <cit.>. Recent studies have also employed KD in various recommendation tasks, such as click-through rate (CTR) prediction <cit.>, topology distillation <cit.>, and multi-task transfer <cit.>.
Prior theoretical analyses <cit.> have indicated that the teacher's predictions, also referred to as soft labels, are typically viewed as more informative training signals compared to binary hard labels. KD can adaptively adjust the training dynamics of the student model based on the values of the soft labels. Knowledge distillation is typically employed to transfer knowledge from a larger parameterized model to a smaller one. Recent studies have applied knowledge distillation to transfer knowledge from KNN to parameterized models <cit.>, demonstrating the potential of knowledge distillation in non-parametric models.





§ CONCLUSION




In current recommendation systems, temporal data shift poses a significant challenge.
The presence of data shift prevents the system from simply enhancing the CTR model's adaptability to new data by adding more training data.
We observed that although the correlation between features and labels in recommendation systems changes over time, if a fixed search space is established, the relationship between the data and the search space remains invariant.
Therefore, we designed a framework that uses retrieval techniques to leverage shifting data for training a relevance network.
However, due to the use of BM25 as a retrieval method, this framework is challenging to deploy in online recommendation systems. 
We then designed a distillation method using knowledge distillation to transfer knowledge from the relevance network to a parameterized module, the search-distill module. We refer to this entire process as the Retrieval and Distill paradigm (RAD).
With the RAD paradigm, we have an effective method for leveraging shifting data to enhance the performance of CTR models.
In future research directions, we aim to incorporate a wider variety of data into the CTR model using RAD. On the other hand, enhancing the performance of the distillation method is also a significant area of focus.


To Robert, for the bagels and explaining CMYK and color spaces.


ACM-Reference-Format


