Federated User Representation Learning
Duc Bui Kshitiz Malik Jack Goetz Honglei Liu Seungwhan Moon Anuj Kumar Kang G. Shin



Fast shared response model for fMRI data
    [
    December 30, 2023
========================================




Collaborative personalization, such as through learned user representations (embeddings), can improve the prediction accuracy of neural-network-based models significantly. We propose Federated User Representation Learning (), a simple, scalable, privacy-preserving and resource-efficient way to utilize existing neural personalization techniques in the Federated Learning (FL) setting.  divides model parameters into federated and private parameters. Private parameters, such as private user embeddings, are trained locally, but unlike federated parameters, they are not transferred to or averaged on the server. We show theoretically that this parameter split does not affect training for most model personalization approaches. Storing user embeddings locally not only preserves user privacy, but also improves memory locality of personalization compared to on-server training. We evaluate FURL on
two datasets, demonstrating a significant improvement in model quality with 8% and 51% performance increases, and approximately the same level of performance as centralized training with only 0% and 4% reductions. Furthermore, we show that user embeddings learned in FL and the centralized setting have a very similar structure, indicating that can learn collaboratively through the shared parameters while preserving user privacy.
























































§ INTRODUCTION


Collaborative personalization, like learning user embeddings 
jointly with the task, is a powerful way to improve accuracy 
of neural-network-based models by adapting the model to each 
user's behavior <cit.>. 
However, model personalization usually assumes the availability of user data 
on a centralized server.
To protect user privacy,
it is desirable to train personalized models in a 
privacy-preserving way, for example, using
Federated Learning <cit.>.
Personalization in FL poses many challenges due to its 
distributed nature, high communication costs, and privacy 
constraints <cit.>.




To overcome these difficulties, we propose a simple,
communication-efficient, scalable, privacy-preserving scheme, 
called , to extend existing neural-network 
personalization to FL.
 can personalize models in FL by learning task-specific 
user representations (i.e., embeddings) <cit.> 
or by personalizing model weights <cit.>.
Research on collaborative personalization in FL <cit.> has generally focused on the development of new techniques tailored to the FL setting. We show that most existing neural-network personalization techniques, which satisfy the constraint (<ref>,<ref>,<ref>), can be used directly in FL, with only a small change to Federated Averaging  <cit.>, the most common FL training algorithm.

Existing techniques do not efficiently train user embeddings in FL since the standard Federated Averaging algorithm <cit.> transfers and averages all parameters on a central server. Conventional training assumes that all user embeddings are part 
of the same model. Transferring all user embeddings to devices during FL training is prohibitively resource-expensive (in terms of communication and storage on user devices) 
and does not preserve user privacy.

 defines the concepts of federated and private parameters: 
the latter remain on the user device instead of being 
transferred to the server.

Specifically, we use a private user embedding vector on each device and train it jointly with the global model.
These embeddings are never transferred back to the server. 

We show theoretically and empirically that splitting model parameters as in affects neither 
model performance nor the inherent structure in learned user embeddings.
While global model aggregation time in increases linearly in the number of users, this is a significant reduction compared with other approaches <cit.> whose global aggregation time increases quadratically in the number of users.

has advantages over conventional on-server training since it exploits the fact that models are already distributed across users. There is little resource overhead in distributing the embedding table across users as well. Using a distributed embeddings table improves the memory locality of both training embeddings and using them for inference, compared to on-server training with a centralized and potentially very large user embedding table. 

Our evaluation of document classification tasks on two real-world datasets shows that has similar performance to the server-only approach while preserving user privacy.
Learning user embeddings improves the performance significantly in both server training and FL.
Moreover, user representations learned in FL have a similar structure to those learned in a central server, indicating that embeddings are learned independently yet collaboratively in FL.



Our contributions: 1) Demonstrate the effectiveness of personalization through learning user representations in FL.
As far as we are aware, this is the first work to do so.
2) The evaluation results show that the personalization improves the performance significantly and
the performance of the FL models are on par with the training on the central server while preserving
user privacy, and 3) analysis of user embeddings learned in both settings shows similar
structures, indicating that the user embeddings are learned independently yet collaboratively.






In this paper, we make the following contributions:


    
  * We propose , a simple, scalable, resource-efficient, and privacy preserving method that enables existing collaborative personalization techniques to work in the FL setting with only minimal changes by splitting the model into federated and private parameters.
    
  * We provide formal constraints under which the parameter splitting does not affect model performance. Most model personalization approaches satisfy these constraints when trained using Federated Averaging <cit.>, the most popular FL algorithm. 
	
  * We show empirically that significantly improves the performance of models in the FL setting. The improvements are 8% and 51% on two real-world datasets. We also show that performance in the FL setting closely matches the centralized training with small reductions of only 0% and 4% on the datasets.
	
  * Finally, we analyze user embeddings learned in FL and compare with the user representations learned in centralized training, showing that both user representations have similar structures.









Our goal is to build an efficient general personalization prediction in FL.
It is not straightforward to just use the existing user embeddings techniques in FL.
The standard Federated Averaging algorithm does not have support for private parameters which are not transferred back to the server but  transfers and averages all parameters.

We design and implement personalization
in Federated Learning (FL) by learning task-specific user embeddings locally on user devices.
We introduce the concepts of federated and private parameters
in which the private parameters stay on the user device and are not transferred back to the server.
Since we leverage the fact that models are already distributed, this method has minimal overhead compared with maintaining a huge user embedding table on the server.
Having user embeddings on each local device is equivalent to have a distributed user embedding table across all users while having minimal cost of maintaining them.




We evaluated the techniques on two datasets: Facebook Messenger Sticker Click-Through Rate and Subreddit datasets. The results show that the personalization in FL has similar performance compared with the server-only approach.




§ RELATED WORK


Most existing work on collaborative personalization in the FL setting has focused on FL-specific implementations of personalization. Multi-task formulations of Federated Learning (MTL-FL) <cit.> present a general way to leverage the relationship among users to learn personalized weights in FL.
However, this approach is not scalable since the number of parameters increases quadratically with the number of users. We leverage existing, successful techniques for on-server personalization of neural networks that are more scalable but less general, i.e., they satisfy the constraint (<ref>,<ref>,<ref>). 


Transfer learning has also been proposed for personalization in FL <cit.>, but it requires alternative freezing of local and global models, thus complicating the FL training process. 
Moreover, some versions <cit.> need access 
to global proxy data. <cit.> uses a two-level 
meta-training procedure with a separate query set to 
personalize models in FL. 

 is a scalable approach to collaborative personalization that does not require complex multiphase training, works empirically on non-convex objectives, and leverages existing techniques used to personalize neural networks in the centralized setting. We show empirically that user representations learned by  
are similar to the centralized setting. 
Collaborative filtering <cit.> can be seen as a 
specific instance of the generalized approach in . Finally, while fine-tuning individual user models after FL training <cit.> can be effective, we focuses 
on more powerful collaborative personalization that leverages common behavior among users.



Ammad-ud-din et al. <cit.> introduce federated collaborative filtering (CF) in which user devices send gradient updates of the factor matrix instead of the matrix values.




However, their method does not generalize for arbitrary inputs like messages or command.
Our method works for general prediction rather than only for recommendation systems.
 a technique using personalization vector and transfer learning. However, the technique requires the freezing of the local and global model and does not provide comprehensive evaluation to compare with an equivalent personalized technique trained on the server.
In this paper, we introduce a simple notion of federated and private parameters so that the learning process does not need to use the complex parameter freezing.





Mocha <cit.> solves learning challenges in FL by formulating FL as a convex multi-task learning to leverage the relationship among users to learn personalized models for each user.

	However, Mocha is restricted in convex regularized linear models.
	Caldas et al. <cit.> introduces a kernelized version of Mocha but lacks of evaluation on practical datasets.
	learns user embeddings explicitly to leverage existing neural models and requires minimal changes to FL.


Federated Meta Learning <cit.> uses a two-level meta-training procedure and requires a separate query set.

Hartman <cit.> uses transfer learning for personalization in FL but it requires alternative freezing of local and global models, thus complicating the FL training process.
Neither it provides any comparison with an equivalent server-trained model.
We introduce a simple but effective technique, , with very few changes to the FL training algorithm and also show that it performs very close to on-server training. 




§ LEARNING PRIVATE USER REPRESENTATIONS

The main constraint in preserving privacy while learning user embeddings is that embeddings should not be transferred back to the server nor distributed to other users.
While typical model parameters are trained on data from all users, user embeddings are very privacy-sensitive <cit.> because a user's embedding is trained only on that user's data.

 proposes splitting model parameters into federated and private parts. In this section, we show that this parameter-splitting has no effect on the FL training algorithm, as long as the FL training algorithm satisfies the constraint. Models using common personalization techniques like collaborative filtering, personalization via embeddings or user-specific weights satisfy the constraint when trained using Federated Averaging.



 §.§ Constraint








FL algorithms typically have two steps: 

	
  * Local Training: Each user initializes their local model parameters to be the same as the latest global parameters stored on the server. Local model parameters are then updated by individual users by training on their own data. This produces different models for each user.[Although not required, for expositional clarity we assume local training uses gradient descent, or a stochastic variant of gradient descent.] 
	
  * Global Aggregation: Locally-trained models are "aggregated" together to produce an improved global model on the server. Many different aggregation schemes have been proposed, from a simple weighted average of parameters <cit.>, to a quadratic optimization <cit.>.


To protect user privacy and reduce network communication, user embeddings are treated as private parameters and not sent to the server in the aggregation step. Formal conditions under which this splitting does not affect the model quality are described as follows.

Suppose we train a model on data from n users, and the k-th training example for user i has features x_k^i, and label y_k^i. The predicted label is ŷ_k^i = f(x_k^i; w_f, w_p^1,…,w_p^n), where the model has federated parameters w_f ∈ℝ^f and private parameters w_p^i ∈ℝ^p  ∀ i ∈{1,…,n}.

In order to guarantee no model quality loss from splitting
of parameters,  requires the constraint to be satisfied, i.e., any iteration of training produces the same results irrespective of whether private parameters are kept locally, or shared with the server in the aggregation step. The two following constraints are sufficient (but not necessary) to satisfy the constraint: local training must satisfy the constraint (<ref>), and global aggregation must satisfy the constraint (<ref>,<ref>).




 §.§ Constraint


The constraint requires that the loss function used in local training on user i is independent of private parameters for other users w_p^j, ∀ j ≠ i. A corollary of this constraint is that for training example k on user i, the gradient of the local loss function with respect to other users' private parameters is zero: 

    ∂ L(ŷ_k^i, y_k^i)/∂w_p^j = ∂ L(f(x_k^i; w_f, w_p^1,…,w_p^n), y_k^i)/∂w_p^j = 0  ,   ∀ j ≠ i


zero_grad_eqn is satisfied by most implementations 
of personalization techniques like collaborative filtering, personalization via user embeddings or user-specific model weights, and MTL-FL <cit.>. Note that (<ref>) is not satisfied if the loss function includes a norm of the global user representation matrix for regularization. In the FL setting, global regularization of the user representation matrix is impractical from a bandwidth and privacy perspective. Even in centralized training, regularization of the global representation matrix slows down training a lot, and hence is rarely used in practice <cit.>. Dropout regularization does not violate (<ref>). Neither does regularization of the norm of each user representation separately. 



 §.§ Constraint


The constraint requires, informally, that the global aggregation update for any parameter depends only on locally trained values of the same parameter, and (optionally), on some summary statistics from training data (e.g, the number of training examples per user). In particular, it implies that the aggregation step has no interaction terms between private parameters of different users. Since interaction terms increase quadratically in the number of users, scalable FL approaches, like Federated Averaging and its derivatives <cit.> satisfy the assumption. However, MTL-FL versions <cit.> do not.  


Given the global parameter w ∈ℝ (federated or private) at the end of iteration t as w^t+1, to satisfy the constraint, the global update rule must be of the form:

    w^t+1 = a(w^t, w_1^t,…,w_n^t, s_1,…,s_n)

where w_i^t is the local update of w from user i, s_i ∈ℝ^s is summary information about training data of user i that is independent of model parameters (e.g., 
the number of training examples), and a is a function from ℝ^1+n+n*s↦ℝ, 

It is worth noting that (<ref>) implies that the global aggregation step cannot perform any norm regularization. We remove this restriction in a relaxed version of the constraint in Section <ref>.


The constraint requires, informally, that the global update step for federated parameters is independent of private parameters.
In particular, the global update for federated parameters depends only on locally trained values of federated parameters, and optionally, on some summary statistics of training data.

Furthermore, the global update step for private parameters for user i is required to be independent of private parameters of other users, and independent of the federated parameters. 
The global update for private parameters for user i depends only on locally trained values of private parameters for user i, and optionally, on some summary statistics.

The constraint implies that the aggregation step has no interaction terms between private parameters of different users. Since interaction terms increase quadratically in the number of users, scalable FL approaches, like Federated Averaging and its derivatives <cit.> satisfy the assumption. However, MTL-FL formulations <cit.> do not. 

More formally, at the beginning of training iteration t, let w_f^t ∈ℝ^f denote federated parameters and w_p^i,t∈ℝ^p denote private parameters for user i. These are produced by the global aggregation step at the end of the training iteration t-1. 



  
Local Training At the start of local training iteration t, model of user i initializes its local federated parameters as u_f,i^t := w_f^t, and its local private parameters as u_p,i^i,t := w_p^i,t, where u represents a local parameter that will change during local training.
u_p,i^i,t denotes private parameters of user i stored locally on user i's device.
Local training typically involves running a few iterations of gradient descent on the model of user i, which updates its local parameters u_f,i^t and u_p,i^i,t.



  
Global Aggregation At the end of local training, these locally updated parameters u are sent to the server for global aggregation. indep_agg_federated_eqn for federated parameters and indep_agg_private_eqn for private parameters must hold to satisfy the constraint. In particular, the global update rule for federated parameters w_f ∈ℝ^f must be of the form:

    w_f^t+1 := a_f(w_f^t, u_f,1^t,…,u_f,n^t, s_1,…,s_n)

where u_f,i^t is the local update of w_f from user i in iteration t, s_i ∈ℝ^s is summary information about training data of user i (e.g., number of training examples), and a_f is a function from ℝ^f+nf+ns↦ℝ^f. 

Also, the global update rule for private parameters of user i, w_p^i ∈ℝ^p, must be of the form:

    w_p^i,t+1 := a_p(w_p^i,t, u_p,i^i,t,s_1,…,s_n)

where u_p,i^i,t is the local update of w_p^i from user i in iteration t, s_i ∈ℝ^s is summary information about training data of user i, and a_p is a function from ℝ^2p+ns↦ℝ^p. 

In our empirical evaluation of , we use Federated Averaging as the function a_f, while the function a_p is the identity function w_p^i,t+1 := u_p,i^i,t (more details in Section <ref>). However, 's approach of splitting parameters is valid for any FL algorithm that satisfies (<ref>) and (<ref>). 



 §.§ with Federated Averaging


works for all FL algorithms that satisfy the constraint. Our empirical evaluation of uses Federated Averaging <cit.>, the most popular FL algorithm. 

The global update rule of vanilla Federated Averaging satisfies the constraint since the global update of parameter w after iteration t is:

    w^t+1 = ∑_i=1^nu_i^tc_i/∑_i=1^nc_i


where c_i is the number of training examples for user i, and u_i^t is the value of parameter w after local training on user i in iteration t. Recall that u_i^t is initialized to w^t at the beginning of local training. 
Our implementation uses a small tweak to the global update rule for private parameters to simplify implementation, as described below.


In practical implementations of Federated Averaging <cit.>, instead of sending trained model parameters to the server, user devices send model deltas, i.e., the difference between the original model downloaded from the server and the locally-trained model: d_i^t := u_i^t - w^t, or u_i^t = d_i^t + w^t. Thus, the global update for Federated Averaging in fed_avg can be written as:

    w^t+1 = ∑_i=1^n (d_i^t + w^t)c_i/∑_i=1^nc_i = ∑_i=1^n d_i^t c_i + ∑_i=1^n w^t c_i/∑_i=1^nc_i = w^t + ∑_i=1^nd_i^tc_i/∑_i=1^nc_i


Since most personalization techniques follow zero_grad_eqn, the private parameters of user i, w_p^i don't change during local training on other users. Let d_p,i^j,t be the model delta of private parameters of user i after local training on user j in iteration t, then d_p,i^j,t= 0, ∀ j ≠ i. fed_avg_delta for updating private parameters of user i can hence be written as:

    w_p^i,t+1 = w_p^i,t + ∑_j=1^nd_p,i^j,tc_j/∑_i=1^nc_i = w_p^i,t + d_p,i^i,tc_i/∑_i=1^nc_i = w_p^i,t + z_i d_p,i^i,t


where z_i = c_i/∑_i=1^nc_i.

The second term in the equation above is multiplied by a noisy scaling factor z_i, an artifact of per-user example weighting in Federated Averaging. While it is not an essential part of , our implementation ignores this scaling factor z_i for private parameters. Sparse-gradient approaches for learning representations in centralized training <cit.> also ignore a similar scaling factor for efficiency reasons. Thus, for the private parameters of user i, we simply retain the value after local training on user i (i.e., z_i = 1) since it simplifies implementation and does not affect the model performance: 


    w_p^i,t+1 = w_p^i,t + d_p,i^i,t = u_p^i,t

where u_p^i,t is the local update of w_p^i from user i in iteration t. In other words, the global update rule for private parameters of user i is to simply keep the locally trained value from user i. 





 §.§ Training Process


While this paper focuses on learning user embeddings, our
approach is applicable to any personalization technique 
that satisfies the constraint. 
The training process is as follows:


	
  * Local Training: Initially, each user downloads the latest federated parameters from the server. Private parameters of user i, w_p^i are initialized to the output of local training from the last time user i participated in training, or to a default value if this was the first time user i was trained. Federated and private parameters are then jointly trained on the task in question. 
	
  * Global Aggregation: Federated parameters trained in the step above are transferred back to, and get averaged on the central server as in vanilla Federated Averaging. Private parameters (e.g., user embeddings) trained above are stored locally on the user device without being transferred back to the server. These will be used for the next round of training. They may also be used for local inference. 

Figure <ref> shows 
an example of federated and private parameters, explained further in Section <ref>

Figure <ref> shows 
an example of federated and private parameters in
a model used for document classification in our empirical evaluation. The model includes a Bidirectional LSTM (BLSTM) layer feeding into a Multi-Layer Perceptron (MLP) - all the parameters in these layers are federated parameters that are shared across all users. These parameters are locally trained and sent back to the server and averaged as in standard Federated Averaging. Another input to the MLP is a user embedding - this embedding is considered a private parameter. It is jointly trained with federated parameters, but kept privately on the device. Even though user embeddings are trained independently on each device, they evolve collaboratively through the globally shared model, i.e., embeddings are multiplied by the same shared model weights.
The federated parameters of the Bidirectional LSTM (BLSTM) and Multi-Layer Perceptron (MLP) layers are locally trained and sent back to the server and averaged as in standard Federated Averaging while the private user embeddings are jointly trained and privately kept on the device. 




§ EVALUATION


We evaluate the performance of on two document classification tasks that reflect real-world data distribution across users.  



 §.§ Experimental Setup






  
Datasets
We use two datasets, called Sticker and Subreddit. Their characteristics are as follows. 



	
  * In-house production dataset (Sticker): This proprietary dataset from a popular messaging app has randomly selected, anonymized messages for which the app suggested a Sticker as a reply. The features are messages; the task is to predict user action (click or 
	not click) on the Sticker suggestion, i.e., binary classification. 
	The messages were automatically collected, de-identified, and annotated; they were not read or labeled by human annotators.
	
	
    
	
  * Reddit comment dataset (Subreddit): These are user comments on the top 256 subreddits on reddit.com.
	Following <cit.>, we filter out users who have fewer than 150 or more than 500 comments, so that each user has sufficient data.
	The features are comments; the task is to predict the subreddit where the comment was posted, i.e., multiclass classification.
	The authors are not affiliated with this publicly available dataset <cit.>. 



Sticker dataset has 940K samples and 3.4K users (274 messages/user on average) while Subreddit has 942K samples and 3.8K users (248 comments/user on average).
Each user's data is split (0.8/0.1/0.1) to form train/eval/test sets.
Table <ref> presents the summary statistics 
of the datasets.






  
Document Model

We formulate the problems as document classification tasks and use the a LSTM-based <cit.> neural network architecture in Figure <ref>.
The text is encoded into an input representation vector by using character-level embeddings and a Bidirectional LSTM (BLSTM) layer.
A trainable embedding layer translates each user ID into a user embedding vector.
Finally, an Multi-Layer Perceptron (MLP) produces the prediction from the concatenation of the input representation and 
the user embedding. All the parameters in the character embedding, BLSTM and MLP layers are federated parameters that are shared across all users. These parameters are locally trained and sent back to the server and averaged as in standard Federated Averaging. User embedding is considered a private parameter. It is jointly trained with federated parameters, but kept privately on the device. Even though user embeddings are trained independently on each device, they evolve collaboratively through the globally shared model, i.e., embeddings are multiplied by the same shared model weights.



  
Configurations
We ran 4 configurations to evaluate the performance of the 
models with/without FL and personalization: Global Server, Personalized Server, Global FL, and Personalized FL. Global is a synonym for non-personalized, Server is a synonym for centralized training. 
The experiment combinations are shown in Table <ref>.



  
Model Training
Server-training uses SGD, while FL training uses Federated Averaging to combine SGD-trained client models <cit.>. Personalization in FL uses training  as described in section <ref>
The models were trained for 30 and 40 epochs for the Sticker and Subreddit datasets, respectively. One epoch in FL means the all samples in the training set were used once.


We ran hyperparameter sweeps to find the best model architectures (such as user embedding dimension, BLSTM and MLP dimensions) and learning rates. 
The FL configurations randomly select 10 users/round and run 1 epoch locally for each user in each round.
Separate hyperparameter sweeps for FL and centralized training resulted in the same optimal embedding dimension for both configurations. The optimal dimension was 4 for the Sticker task and 32 for the Subreddit task. 





  
Metrics
We report accuracy for experiments on the Subreddit dataset.
However, we report AUC instead of accuracy for the Sticker dataset since classes are highly unbalanced. 
Note: FL and server models likely have different hidden dimensions because we also searched these hyperparameters for FL models. Okay, We should try to fix this if paper is accepted.
	
















 §.§ Evaluation Results




  
Personalization improves the performance significantly.
User embeddings increase the AUC on the Sticker dataset by 7.85% and 8.39% in the Server and FL configurations, respectively.
The improvement is even larger in the Subreddit dataset with 37.2% and 50.51% increase in the accuracy for the Server and FL settings, respectively.
As shown in Figure <ref>, these results demonstrate that the user representations effectively learn the features of the users from their data.




  
Personalization in FL provides similar performance as server training.

There is no AUC reduction on the Sticker dataset while the accuracy drops only 3.72% on the Subreddit dataset (as shown in Figure <ref>).
Furthermore, the small decrease of FL compared to centralized training is expected and consistent with other results <cit.>.
The learning curves on the evaluation set on Figure <ref> show the performance of FL models
asymptotically approaches the server counterpart.
Therefore, FL provide similar performance with the centralized setting while protecting the user privacy.






  
User embeddings learned in FL have a similar structure to those learned in server training.
Recall that for both datasets, the optimal embedding dimension was the same for both centralized and FL training. We visualize the user representations learned in both the centralized and FL settings using t-SNE <cit.>. 
The results demonstrate that similar users are clustered together in both settings. 


Visualization of user embeddings learned in the Sticker dataset in Figure <ref> shows that users having similar (e.g., low or high) click-through rate (CTR) on the suggested stickers are clustered together.
For the Subreddit dataset, we highlight users who comment a lot on a particular subreddit, for the top 5 subreddits (AskReddit, CFB, The_Donald, nba, and politics). Figure <ref> indicates that users who submit their comments to the same subreddits are clustered together, in both settings.
Hence, learned user embeddings reflect users' subreddit commenting behavior, in both FL and Server training.










Experiments to do:

    
  * Measure performance of the server-side personalization model on the datasets.
    
  * Measure performance of the FL personalization model on the the datasets.
    
  * Compare the results of the user embedding learned in FL and the central server settings. Expect that personalized FL has minimal degradation compared with the training on the central server on all tasks.
    
  * Analyze user embeddings learned in the FL and central server settings. Expect that user embeddings learned in FL have characteristics similar to ones learned on the central server.
	
  * Show that personalization learns high-dimensional representations to improve the prediction accuracy.


	















§ CONCLUSION AND FUTURE WORK

This paper proposes , a simple, scalable, bandwidth-efficient technique for model personalization in FL.  improves performance over non-personalized models and achieves similar performance to centralized personalized model while preserving user privacy.
In this paper, we introduced the first implementation of personalization in Federated Learning 
which improves performance over non-personalized models 
and achieves similar performance to centralized personalized 
model while preserving the user privacy. 
Moreover, representations learned in both server training and FL show similar structures.
In future, we would like to evaluate  on other datasets and models, learn user embeddings jointly across multiple tasks, 
address the cold start problem
and personalize for users not participating in global FL aggregation.










plainnat







