PoisonedFL: Model Poisoning Attacks to Federated Learning via Multi-Round Consistency
Yueqi Xie Minghong Fang Neil Zhenqiang Gong







  HKUST
  
  
  
  
  

yxieay@connect.ust.hk



  Duke University
  
  
  
minghong.fang@duke.edu



  Duke University
  
  

neil.gong@duke.edu






 Model poisoning attacks are critical security threats to Federated Learning (FL). Existing model poisoning attacks suffer from two key limitations: 1) they achieve suboptimal effectiveness when defenses are deployed, and/or 2) they require knowledge of the model updates or local training data on genuine clients. 
In this work, we make a key observation that their suboptimal effectiveness arises from only leveraging model-update consistency among malicious clients within individual training rounds, making the attack effect self-cancel across training rounds. In light of this observation, we propose , which enforces multi-round consistency among the malicious clients' model updates while  not requiring any knowledge about the genuine clients. 
Our empirical evaluation on five benchmark datasets shows that  breaks eight state-of-the-art defenses and outperforms seven existing model poisoning attacks. Moreover, we also explore new defenses that are tailored to , but our results show that we can still adapt  to break them. Our study shows that FL systems are considerably less robust than previously thought, underlining the urgency for the development of new defense mechanisms. 




















PoisonedFL: Model Poisoning Attacks to Federated Learning via Multi-Round Consistency
    Neil Zhenqiang Gong
    
=====================================================================================





§ INTRODUCTION


Federated Learning (FL) enables clients–such as smartphones, self-driving cars, and IoT devices–to collaboratively train a model without sharing their raw local training data with a central server <cit.>. 
Typically, FL involves multiple training rounds, each of which consists of three key steps: 1) the server sends the current global model to the clients or a subset of them; 2) a client trains a local model based on the current global model and its local training data, and sends the model update (i.e., the difference between local model and global model) to the server; and 3) the server aggregates the clients' model updates according to an aggregation rule and adds the aggregated model update to the global model.  
 FL has wide applications across diverse domains, including healthcare <cit.> and finance <cit.>. 


However, due to the distributed nature, FL is fundamentally vulnerable to model poisoning attacks <cit.>. In particular, an attacker can control some malicious clients  and send carefully crafted malicious model updates to the server to poison the global model. The malicious clients could be compromised  genuine clients  or injected fake clients. The poisoned global model misclassifies many inputs indiscriminately, i.e., it has a large testing error rate (also known as untargeted model poisoning attacks). 


Existing model poisoning attacks suffer from two key limitations. First, they <cit.> only leverage in-round consistency, 
i.e.,  they send the same or similar malicious model updates in a single training round to amplify the attack effect, but neglect multi-round consistency.
For instance, Fang attack <cit.> crafts the malicious model updates in a given training round to maximally deviate the global model in that round, 
while MPAF <cit.> sets the malicious model updates in a training round as the magnified difference between the current global model and a random model.
 Such a strategy limits the attack effectiveness when defenses are deployed. 
This is because defenses weaken the attack effect in every training round by filtering and clipping the malicious model updates; and the weakened attack effects further self-cancel across training rounds due to the lack of multi-round consistency, as shown by our measurement results in Section <ref>.  Second, most existing attacks <cit.> require knowledge of genuine clients' model updates or local training data. They often assume compromising a significant number of genuine clients to obtain such knowledge, which is notably impractical. 
These attacks with suboptimal effectiveness and/or impractical assumptions give a false sense of FL security. 





Our work
In this work, we propose , a novel model poisoning attack to FL, which overcomes the above limitations of existing attacks.   
Specifically, to enhance attack effectiveness,  leverages two key components: 1) multi-round consistency, and 2) dynamic attack magnitude adjustment. 
The first component ensures that even if the attack effect is weakened in individual training rounds by defenses, the cumulative attack effect over multiple training rounds remains undeterred.
The second component aims to prevent the malicious model updates from being entirely filtered out in individual training rounds by defenses. Moreover,  does not require genuine clients' information, and is  agnostic to the defenses deployed by the server. 

For the first component, our key observation is that a model, when updated substantially in a random direction,  exhibits significantly degraded accuracy (see Fig. <ref> in Appendix). Based on this observation, we formulate an all-round optimization problem that facilitates multi-round consistency.  
Specifically, the objective of our optimization problem is to maximize the magnitude of the total aggregated model updates in a random update direction,  which is characterized by a random sign vector s. Each dimension of the sign vector s is either +1 or -1 uniformly at random, meaning that the total aggregated model updates increase or decrease the corresponding parameter of the global model. With such total aggregated model updates, the final global model is substantially moved along the random update direction, leading to a large testing error rate. Note that our objective stands substantially distinct from those of existing attacks, as it enforces multi-round consistency without self-cancellation. 
However, it is challenging to solve the all-round optimization problem in a given round because the model updates in future training rounds are not available. To address the challenge, we transform the optimization problem as a per-round one, which aims to find malicious model updates on the malicious clients to maximize the magnitude of the aggregated model update along the update direction s in each round. In particular, our malicious model update is a dimension-wise product of the sign vector s and a magnitude vector, a dimension of which indicates the magnitude of the corresponding dimension of the malicious model update.



For the second component, we propose to dynamically adjust the magnitude vector in each training round based on the attack results in the past rounds, making our attack effective against the unknown defense that the server may have deployed.  Our key idea is to find a magnitude vector that has a large magnitude while avoiding the malicious model update being filtered out by the unknown defense. On one hand, based on the global models and malicious model updates in the previous training rounds, we estimate the model updates of genuine clients to determine an unit magnitude vector. The magnitude vector is the product of the unit magnitude vector and a scaling factor.  On the other hand, we set the scaling factor based on whether the attack achieves the objective in the past rounds. In particular, our attack objective is that the total aggregated model updates have the random update direction s. We leverage statistical hypothesis testing to check whether our attack in the previous rounds achieves this objective. If not, it means the malicious updates were filtered out by the defense in previous rounds and thus we decrease the scaling factor to be more stealthy.   



We evaluate our  on five benchmark datasets. Our results show that,  under the least knowledge scenario – that is, no knowledge about the defense nor genuine local training data/models –  can break eight state-of-the-art FL defenses <cit.>. For instance,  can reduce the global models to random guessing in many cases. Moreover, in most cases,  achieves higher testing error rates than existing attacks <cit.> 
when a defense is deployed even if we assume the attacker has access to the local models of all genuine clients. In the remaining cases, both  and some existing attacks reduce the global models to random guessing. Furthermore,  remains effective across different FL settings including the fraction of malicious clients, the degree of non-IID of the genuine clients' local training data, the number of epochs used to train a local model, and the fraction of clients selected to participate in each training round.  

We also explore new defenses tailored to . For instance,  enforces the malicious model updates across training rounds to have the same sign vector, but model updates from genuine clients may not satisfy such constraints. Therefore, we design a method to detect fake clients based on the sign vectors of the model updates across multiple training rounds. However, we can still adapt  to counter such defense with a minor impact on attack effectiveness. In particular, instead of using the same sign vector across all training rounds, our adapted  randomly flips a small portion of the sign vector in each training round when crafting the malicious model updates. Our results highlight the need for new defense mechanisms. 

To summarize, our key contributions are  as follows:

    
  * We uncover a fundamental limitation of existing model poisoning attacks, i.e., their inconsistent malicious model updates across multiple training rounds self-cancel the attack effect, leading to suboptimal attack effectiveness. 
     
  * We propose  that leverages consistent malicious model updates, does not require knowledge about genuine local training data or local models, and is agnostic to the defense deployed by the server.
    
  * We perform extensive experiments to demonstrate that  breaks state-of-the-art FL defenses and outperforms state-of-the-art attacks. 
    
  * We explore new defenses tailored to  and show their insufficiency by adapting  to counter them. 
  



§ BACKGROUND AND RELATED WORK











 §.§ Federated Learning (FL)

Federated Learning (FL) <cit.> is a machine learning paradigm that allows many clients (e.g., smartphones) to collaboratively train a model without sending their raw local training data to a server.  
Suppose we have n clients and each client has a local training dataset 𝒟_i, i=1,2,⋯,n. 
 FL aims to solve an optimization problem that is defined as min∑_i=1^nℒ(w,𝒟_i), 
where w∈ℝ^d is the model (called global model), d is the number of parameters in the model w,  and ℒ(w,𝒟_i) is the local loss function for client i.  
FL iteratively trains the global model w. In the t-th training round, the server first distributes the current global model w^t-1 to all clients or a subset of them.  For simplicity, we assume all clients are selected, but we will also evaluate a subset of them are selected in each training round in our experiments.  
Each client i then trains a local model w_i^t based on the current global model and its local training data. For instance, client i initializes w_i^t as w^t-1 and iteratively updates w_i^t via minimizing  ℒ(w,𝒟_i) using Stochastic Gradient Descent (SGD). After training the local model w_i^t, client i sends the model update g^t_i=w^t_i-w^t-1 to the server. Finally, the server aggregates the received  model updates and updates the global model as follows:

    w^t=w^t-1 + g^t,

where g^t=AR({g^t_i}_i ∈[1, n]) is the aggregated model update, AR denotes the aggregation rule used by the server, and {g^t_i}_i ∈[1, n] is the set of clients' model updates.  For instance, when AR is FedAvg <cit.>, the aggregated model update AR({g^t_i}_i ∈[1, n]) is the average of the clients' model updates. 









 §.§ Poisoning Attacks to FL

In model poisoning attacks <cit.>, malicious clients corrupt the global model via sending carefully crafted malicious model updates to the server. 
In a targeted attack <cit.>, a poisoned global model only misclassifies attacker-chosen inputs with certain properties (e.g., embedded with an attacker-chosen trigger). In contrast, in an untargeted attack <cit.>, a poisoned global model 
indiscriminately misclassifies a large number of clean inputs, leading to a large testing error rate and eventually denial-of-service attacks. Therefore, we focus on untargeted attacks in this work. 

Requiring genuine clients' information One category of model poisoning attacks <cit.> require that an attacker has access to genuine clients' model updates or local training data. To obtain such knowledge, these attacks often assume an attacker can compromise a large fraction of genuine clients. 
For instance, in a little is enough (LIE) attack <cit.>, the attacker computes the average of the genuine model updates of the compromised genuine clients and adds perturbation to them to craft the malicious model updates in each training round. Concurrent to LIE, Fang et al. <cit.> proposed the first optimization-based attack framework that can be tailored to any aggregation rule. 
Their key idea is that the attacker crafts the malicious model updates  such that the aggregated model update after attack deviates substantially from the before-attack one in each training round. 
Building upon this attack framework <cit.>, the authors in <cit.> assume the malicious model update to be the sum of the before-attack aggregated model update and a fixed perturbation vector scaled by a factor, where the scaling factor is optimized to maximize the deviation between the after-attack aggregated model update and the before-attack one. 


Not requiring genuine clients' information The other category of model poisoning attacks <cit.> do not rely on any genuine clients' information. Therefore, they only assume that an attacker can inject fake clients into an FL system, which is more realistic than compromising genuine clients.
Since genuine clients' information is not available, these attacks craft malicious model updates based on heuristics. For instance, in MPAF <cit.>, the attacker crafts malicious model updates such that the global model progressively moves towards a random model in each training round.


Limitations of existing attacks
Existing attacks suffer from the following limitations.
First, the first category of attacks require an attacker to compromise a large fraction of genuine clients, which is not affordable in production FL <cit.>. 
Second, both categories of attacks craft malicious model updates  independently in different training rounds, leading to  self-cancellation of attack effect across training rounds. As a result, they achieve suboptimal attack effectiveness when defenses are deployed. 









 §.§ Defenses against Poisoning Attacks

Defenses against poisoning attacks to FL can be roughly categorized
into Byzantine-robust aggregation rules, provably robust defenses, and malicious clients detection.

Byzantine-robust aggregation rules Instead of using the average of the clients' model updates to update the global model, Byzantine-robust aggregation rules <cit.> typically employ statistical analysis to filter out or clip outlier model updates before aggregating them.   
For instance, 
 Multi-Krum <cit.>  
iteratively selects the model update that is the closest to its nearest neighbor model updates, and  aggregates them. Dimension-wise  Median (or Trimmed Mean) <cit.> uses the median (or trimmed-mean) of the clients' local models for each dimension. 
 Norm Bound <cit.> aims to minimize the impact of potential malicious model updates by clipping the magnitudes of all model updates.
 FLTrust <cit.> assumes the server uses a clean and small dataset to bootstrap trust.
Specifically, in FLTrust <cit.>, the server trains a server model and assigns trust scores to clients by measuring the similarity between the server model update and clients' model updates. Under some assumptions about the learning problem and  clients' local training data distribution, these Byzantine-robust aggregation rules often can theoretically guarantee that the learnt global model does not change much when the number of malicious clients is bounded. However, they are still vulnerable to our attack because these assumptions often do not hold and a bounded change in the parameters of the global model does not guarantee a small change of error rate. 



Provably robust defenses Provably robust defenses <cit.> can provide stronger theoretical guarantees than Byzantine-robust aggregation rules. In particular, given a number of malicious clients, a provably robust defense can guarantee a lower bound of testing accuracy no matter what poisoning attacks the malicious clients use.    
For instance, FLCert <cit.> trains multiple global models, each of which is trained using a subset of  clients. 
Given a testing input, FLCert predicts its label by taking a majority vote among the multiple global models. A major limitation of these provably robust defenses is that they can only tolerate a small number of malicious clients, i.e., the lower bound of testing accuracy reduces to 0 when the number of malicious clients is moderate. As a result, they are still vulnerable to our attack since an attacker can easily inject a large number of fake clients. 





Malicious clients detection These defenses <cit.> aim to detect malicious clients during or after the training process. After detecting malicious clients, the server can further efficiently recover/re-train an accurate global model using the remaining clients <cit.>. 
For instance, FLDetector <cit.> aims  to detect malicious clients by utilizing the inconsistency of a client's model updates across multiple training rounds. Their key idea is that the model updates of a benign client in multiple training rounds are all calculated based on the client's local training data, while the  model updates of a malicious client are crafted without following the learning procedure. As a result, the model updates of a benign client satisfy certain consistency constraints across  training rounds, while those of a malicious client do not, making it possible to distinguish between benign and malicious clients. In our attack, we enforce the malicious model updates of a malicious client to be consistent across training rounds, making FLDetector ineffective. 




§ THREAT MODEL





Attacker's goal
Following previous works  <cit.>,
we consider the goal of an attacker is to poison the training process of FL such that the final learnt global model makes incorrect predictions for a large number of clean inputs indiscriminately. In other words, the poisoned global model has a large testing error rate. Such a global model would eventually be abandoned, leading to denial-of-service attacks <cit.>.    



Attacker's capability
 We assume that the attacker is capable of injecting fake clients into an FL system as malicious clients. We note that this is a practical threat model. Multiple platforms <cit.> can be used to automatically inject fake devices/clients in a distributed system. For instance, when the FL system learns a global model on Android smartphones (i.e., clients), e.g., Google's next-word prediction on Android devices <cit.>, an attacker can use a laptop to simulate fake smartphones via Android emulators <cit.>. 
 
 

Attacker's background knowledge 
We consider the minimal background knowledge scenario, which represents the most realistic threat model. Specifically, the attacker has access to the global models during the training process since the server sends them to the clients including the fake ones. However, the attacker does not have access to genuine local training data nor local models on any genuine client. Moreover, we assume the attacker does not know the defense (e.g., Byzantine-robust aggregation rule, provably robust defense, or detector) deployed by the server.

We note that our threat model is the same as previous fake client based model poisoning attacks <cit.>, which do not require genuine clients' information. 



§ MOTIVATION






We first introduce our key observation about why existing model poisoning attacks achieve suboptimal attack effectiveness, which motivates the design of our  in the next section. We show experimental results on the MNIST dataset with the default parameter settings described in Section <ref>. We assume the server uses Trimmed Mean <cit.> as the defense. Moreover, we assume the ratio of fake clients to genuine clients is 20%. We consider  MPAF <cit.>, an attack not requiring genuine clients' information, and Fang <cit.>, a representative attack that requires genuine clients' information. 
To give advantages to Fang, we assume the attacker knows the local models of all participating genuine clients in each round when crafting malicious model updates on fake clients. 

In each training round, these attacks craft a malicious model update for each fake client selected by the server to participate in training. A malicious model update g_i^t of fake client i in training round t  can be decomposed into a dimension-wise product of a  sign vector s_i^t and a magnitude vector |g_i^t|, where |·| means dimension-wise absolute value of a vector. Each dimension of s_i^t is either +1 or -1, while all dimensions of |g_i^t| are non-negative. A dimension with a +1 (or -1) sign aims to increase (or decrease) the corresponding dimension/parameter of the global model. For a fake client, we say a dimension is flipped in round t  if its sign in the malicious model update is flipped, compared to that in the previous training round. A flipped dimension in a training round t may cancel the attack effect of the previous training round t-1. This is because the malicious model updates aim to increase the dimension of the global model in one training round but decrease it in the other. For each training round t, we define flipping rate as the fraction of flipped dimensions of a malicious model update averaged over the fake clients. 

Fig. <ref> shows the testing error rate of the global model as a function of training round when no attack, MPAF, or Fang is used; while Fig. <ref> shows the flipping rate as a function of training round when MPAF or Fang is used to craft malicious model updates on the fake clients. We observe that the flipping rate of MPAF is always larger than 35% and that of Fang is larger than 40% in the last around 1,500 training rounds, leading to self-cancellation of attack effect in many rounds. For instance, in Fang attack, the flipping rate is around 10% in nearly the first 1,000 training rounds, i.e., the malicious model updates maintain some degree of “consistency”. As a result, the global model under Fang attack has a large testing error rate in those rounds as shown in Fig. <ref>. However, the flipping rate rapidly increases as the training proceeds, eventually leading to self-cancellation of attack effect and an accurate global model.   




§ 









 §.§ Overview


Our  aims to craft malicious model updates on the fake clients such that the final global model is substantially moved along a random update direction, leading to a large testing error rate.   Towards this goal, our malicious model updates on the fake clients are a dimension-wise product between a sign vector (characterizing the update direction of each dimension) and a magnitude vector (characterizing the update size of each dimension). The sign vector is randomly picked at the beginning and keeps the same across training rounds to enforce consistency; while the magnitude vector dynamically changes to simultaneously have large magnitudes and avoid being filtered out by the unknown defense that may be deployed by the server. In particular, we  set the magnitude vector in a training round by considering whether our attack is successful in the previous training rounds to avoid being filtered out by a defense.  Thus,  the aggregated model updates across training rounds are likely to have consistent sign vectors (i.e., update direction) and add up instead of cancelling out with each other. As a result,  the final global model is substantially moved along the random update direction, leading to a large testing error rate. 

Fig. <ref> illustrates how the global model evolves in existing attacks and our attack in three training rounds, where w^0, w^1, w^2, and w^3 are the global models, while g^1, g^2, and g^3 are the aggregated model updates in the three training rounds.  The aggregated model updates self-cancel across training rounds in existing attacks due to inconsistent update directions (i.e., sign vectors), while they add up towards the same, random update direction in our attack.  




 §.§ Formulating an Optimization Problem 


All-round optimization problem Suppose we have n genuine clients and m fake clients. The genuine clients are numbered as 1, 2, ⋯, n, while the fake clients are numbered as n+1,⋯, n+m.  We denote by T the total number of training rounds and w^T the final global model. Recall that our attack goal is to poison the training process such that the final global model w^T  has a large testing error rate. We observe that w^T=w^0 +∑_t=1^T g^t, where w^0 is the initial global model and g^t=AR({g_i^t}_i∈ [1,n+m]) is the aggregated model update in training round t. We denote by sign(∑_t=1^T g^t) the sign vector (i.e., update direction) of the total aggregated model update ∑_t=1^T g^t in the T training rounds. Each dimension of a sign vector is either +1 or -1, which means increasing or decreasing the corresponding parameter of the global model. 



If the total aggregated model update ∑_t=1^T g^t has a large magnitude and a random update direction, then the final global model w^T is substantially moved along the random update direction  and would have a large testing error rate no matter what the initial global model  w^0 is. For instance, Fig. <ref> in Appendix shows an example where a global model becomes nearly random guessing when moved along a random update direction substantially. Based on this observation, our goal is to craft the malicious model updates {g_i^t}_i∈ [n+1,n+m], t∈ [1,T] on the fake clients in the T training rounds such that the total aggregated model update ∑_t=1^T g^t has a large magnitude along a random update direction. Formally, we formulate the following all-round optimization problem:

    max _g_i^t, i∈[n+1, n+m], t ∈[1, T]∑_t=1^T g^t 
    
    s.t. sign(∑_t=1^T g^t) =s,

where · is ℓ_2-norm and s is a random sign vector that characterizes a random update direction. Each dimension of s is picked as either +1 or -1 uniformly at random. However, the all-round optimization problem is hard to solve. In particular, when solving the malicious model updates in training round t, the model updates in the future training rounds  are not available yet. To address the challenge, we transform our all-round optimization problem into a per-round one. 




Per-round optimization problem Our key idea is to enforce the aggregated model update g^t in each training round t to have the sign vector  s. In particular, for each training round t, we reformulate a per-round optimization problem as follows:

    max _g_i^t, i∈[n+1, n+m]g^t , s.t. sign(g^t) = s.

Since the aggregated model updates across training rounds have the same sign vector s, their magnitudes add up in the total aggregated model update, leading to a bad global model.  





 §.§ Solving the Optimization Problem



It is still challenging to solve the per-round optimization problem in Equation <ref>. This is because  g^t=AR({g_i^t}_i∈ [1,n+m]), but the attacker does not have access to the aggregation rule AR nor the model updates of the genuine clients. We address the challenge via proposing a method to approximately solve the optimization problem. Our key idea is to enforce that the malicious model updates on the fake clients have the same sign vector s, so the aggregated model updates are  likely to have the sign vector s (see Table <ref> in Appendix for an example in our experiments).  Specifically, we assume that the malicious model update g_i^t of fake client i in training round t has the following form:

    g_i^t =k^t ⊙s, ∀ i∈ [n+1, n+m], ∀ t∈ [1,T],

where k^t is the magnitude vector of the malicious model updates in training round t. 

Recall that the objective function of our per-round optimization problem is to maximize the magnitude of the aggregated model update g^t. One naive way to achieve this objective is to select a large magnitude vector k^t. However, such a magnitude vector k^t may be filtered out by the server when a defense (e.g.,  Multi-Krum <cit.>,  FLTrust <cit.>, or FLAME <cit.>) is deployed, leading to suboptimal attack effectiveness as  our experiments in Section <ref> show. Therefore, our goal is to maximize the magnitude vector k^t while avoiding being filtered out by an unknown defense that may be deployed by the server. To achieve this goal, our key idea is to pick k^t that has a large magnitude but is also similar to the model updates on the genuine clients. In particular, we explicitly decompose the magnitude vector k^t into the following form:

    k^t = λ^t ·v^t,

where λ^t is a scaling factor that governs the overall magnitude of k^t, and v^t is a unit magnitude vector used to control the relative magnitudes of different dimensions.  v^t has an ℓ_2-norm of 1. Such decomposition enables us to explicitly solve   v^t and  λ^t, which we discuss next.  



Dynamically adjusting the unit magnitude vector v^t A straightforward approach to pick the unit magnitude vector v^t is to set each dimension to the same value, which means that we treat all dimensions equally.  
However, as our experiments in Section <ref> show, this strategy achieves suboptimal attack effectiveness. This is because the dimensions of the model updates on the genuine clients are unlikely to have the same magnitude. As a result, the model updates of the fake clients are dissimilar to those of the genuine clients, making them filtered out when a defense is deployed on the server. It is challenging to construct a unit magnitude vector similar to those of the genuine model updates, since the attacker does not have access to the model updates on the genuine clients in our threat model. 

To address this challenge, we observe that the global model difference w^t-1-w^t-2 is the aggregated model update g^t-1 in the previous training round t-1. Moreover, g^t-1 is an aggregation of the model updates on the genuine clients and those on the fake clients. The attacker knows w^t-1, w^t-2, and the model updates g_i^t-1=k^t-1⊙s on the fake clients in the previous training round. Therefore, the attacker can estimate the unit magnitude vector of the genuine clients in the previous training round and use it as v^t in the current training round. In particular, we can normalize the model update k^t-1⊙s on the fake clients in the previous training round to have the same magnitude as g^t-1 and then subtract it from g^t-1 to estimate the model updates on the genuine clients. Formally, we have the following:

    v^t = |g^t-1 -  g^t-1/k^t-1⊙s·k^t-1⊙s|  /g^t-1 -  g^t-1/k^t-1⊙s·k^t-1⊙s,

where g^t-1=w^t-1 -w^t-2 is the aggregated model update in the previous training round,  w^t-1 -w^t-2/k^t-1⊙s is used to normalize k^t-1⊙s to have the same magnitude as g^t-1 so they are on the same scale for subtraction, the absolute value |·| is used in the numerator because v^t is a magnitude vector, and the denominator is used to normalize v^t to have ℓ_2-norm of 1.  


Dynamically adjusting the scaling factor λ^t A naive approach to pick λ^t is to set it as a large number, which means using model updates with large magnitudes on the fake clients. However, as our experiments in Section <ref> show, this approach achieves suboptimal attack effectiveness because the malicious model updates may be filtered out by a defense. To address the challenge, we dynamically set λ^t based on the global model difference (i.e., aggregated model update) in the previous training round. Our intuition is that, if the aggregated model update in the previous training round has a large (or small) magnitude, the server may also expect an aggregated model update with a large (or small) magnitude in the current training round. Based on this intuition, we set λ^t to be proportional to the magnitude of the aggregated model update (i.e., w^t-1 -w^t-2) in the previous training round. Formally, we have the following:

    λ^t = c^t  ·w^t-1 -w^t-2,

where c^t is a factor. 

Dynamically adjusting c^t based on hypothesis testing We adjust c^t in training round t based on whether the attack is successful in the past e training rounds. 
Specifically, we formulate the following two hypothesis:

    
  * Null hypothesis H_0. The attack is not successful in the past e training rounds. 
    
  * Alternative hypothesis H_1. The attack is successful in the past e training rounds. 

If the hypothesis testing favors H_1 with a p-value p,  c^t does not change, i.e., we set c^t=c^t-1; otherwise the malicious model updates may have been filtered out by the defense in the past training rounds and thus we decrease c^t by a factor, i.e., c^t=β c^t-1, where β < 1. 
Specifically, the attacker can calculate the total aggregated model update in the past e training rounds, i.e., w^t-1 - w^t-e. Moreover, the attacker can calculate the number (denoted as X) of dimensions of w^t-1 - w^t-e whose signs match with those in s. 
Under the null hypothesis H_0,  the malicious model updates were filtered out by the defense, and thus the signs of the total aggregated model update w^t-1 - w^t-e are not correlated with   s. Since s is picked uniformly at random, X is a random variable and follows a binomial distribution Bin(d, 0.5), where d is the number of dimensions of the global model. Therefore, if we have Pr(x≥ X)≤ p, then we reject H_0 and favor H_1,  where x∼ Bin(d, 0.5) and p is the p-value. We use p=0.01 in experiments.  





§ EVALUATION





 §.§ Experimental Setup




  §.§.§ Datasets

We use five datasets from different domains, including MNIST <cit.>, FashionMNIST <cit.>, Purchase <cit.>, CIFAR-10 <cit.>, and FEMNIST <cit.>.
Details are deferred to Appendix <ref>. 



  §.§.§ Evaluation Metric


We use the testing error rate of the global model as the evaluation metric, which is the fraction of testing inputs that are misclassified by the global model. 
A larger testing error rate indicates a  better attack. 



  §.§.§ Defenses and Compared Attacks


We evaluate eight state-of-the-art defenses, including six Byzantine-robust aggregation rules (Multi-Krum <cit.>, Median <cit.>, Trimmed Mean <cit.>, Norm Bound <cit.>, FLTrust <cit.>, and FLAME <cit.>), one provably robust defense (FLCert <cit.>) and one malicious clients detection method (FLDetector <cit.>). We also consider the non-robust FedAvg <cit.> as a baseline. 
We compare our   with seven attacks, including five attacks (Fang <cit.>, Opt. Fang <cit.>, LIE <cit.>, Min-Max <cit.>, and Min-Sum <cit.>) that require genuine clients' information and two attacks (Random <cit.> and MPAF <cit.>) that do not require such information. 
Details are deferred to Appendix <ref> and <ref>.






  §.§.§ Evaluation Metric

We use the testing error rate of the global model as the evaluation metric, which is the fraction of testing inputs that are misclassified by the global model. 
A larger testing error rate indicates a  better attack. 





  §.§.§ FL Settings

By default, we assume 1,200 genuine clients and 20% of fake clients, i.e., the number of fake clients is 20% of the number of genuine clients. 
In each training round, the server selects a random subset of 10% of clients to participate in training.
A  genuine client trains its local model for 1 local epoch in a training round using SGD. 
Due to limited space, more FL settings for the five datasets are detailed in Appendix <ref>. 
We also study the impact of different FL settings (fraction of fake clients, non-IID degree, local training epochs, and participation rate) in Section <ref>.
Our   has parameters c^t, e, and β. We set c^0=8, e=50, and β=0.7, and we do not decrease c^t to be smaller than 0.5 since a too small c^t is ineffective.  

We conduct our experiments using 16 GeForce RTX 3090 GPUs.











 §.§ Main Results






Table <ref> shows the testing error rates of the learnt global models under different attacks and defenses for the five datasets. “No Attack” means no fake clients are injected. We have the following observations. 





  breaks state-of-the-art FL defenses
  substantially increases the testing error rates of the global models learnt by FedAvg and  state-of-the-art defenses, compared to No Attack.  Specifically,  increases the testing error rate by 15.84 to 90.67 across the five datasets and nine FL methods. Moreover, in 29 out of the 45 cases,   makes the global model nearly random guessing or worse. Our results show that existing FL defenses are not robust even if an attacker has no knowledge about the defense nor the genuine clients' local data or models. 

Why  breaks FL defenses State-of-the-art defenses leverage filtering and clipping to reduce the impact of malicious model updates.  They can weaken the attack effect in individual training rounds.  However, since  crafts malicious model updates that are consistent across training rounds and avoid being entirely filtered out by dynamically adjusting attack magnitudes, its cumulative attack effect still substantially moves the global model towards the designated random update direction s, leading to a large testing error rate. For instance, Table <ref> in Appendix shows that a large fraction (over 90% percent) of the dimensions of the total aggregated model updates have signs matching with s  and Table <ref> in Appendix shows that the magnitude of the total aggregated model updates becomes very large, after our attack effect accumulates over multiple training rounds, for all defenses.  The detailed analysis for each defense is defered to Appendix <ref>.



 outperforms existing attacks 
For the non-robust FedAvg,  both   and some existing attacks can reduce the global models to random guessing or worse. However, for the five datasets and eight FL defenses, our  achieves a much larger testing error rate than existing attacks in 32 out of the 40 cases with the gap ranging from 2.11 to 84.33. In six of the remaining cases, both our  and some existing attacks make the learnt global models nearly random guessing or worse. Our  achieves a smaller testing error rate than some existing attacks only in two cases. In particular,   Fang achieves a testing error rate of 70.02 for the case (Purchase, TrMean), while   achieves a testing error rate of 53.86 in the case; and Min-Max (or Min-Sum)  achieves a testing error rate of 94.88 (or 94.97) for (FEMNIST, FLDetector), while  achieves a testing error rate of 91.25 in the case. However, we note that  still achieves large  testing error rates even in these rare cases. 





Among the three attacks Random, MPAF, and , which do not require knowledge about the local training data/models on the genuine clients nor the defense, Random is  ineffective in most cases when a defense is deployed; while MPAF is effective to  some extent at  attacking some defenses, e.g., TrMean and Norm  Bound, but ineffective at attacking other defenses, e.g., Multi-Krum, FLTrust, and FLDetector.   substantially outperforms Random and MPAF when a defense is deployed in all cases except (CIFAR10, Norm Bound), for which both  MPAF and  make the global models random guessing. For instance, on MNIST dataset,  achieves 59.24 to 85.87 higher testing error rates than MPAF at attacking a defense.  

Why  outperforms existing attacks 
 outperforms existing attacks because their crafted malicious model updates are inconsistent across training rounds, leading to self-cancellation of attack effect. In contrast,  enforces the malicious model updates to be consistent (i.e., have the same sign vector) across training rounds, making the aggregated model updates add up in the global model across training rounds.  As a result, the final learnt global model has a very large magnitude, leading to a large testing error rate. For instance, Table <ref> in Appendix shows the magnitude (measured by ℓ_2 norm) of the total aggregated model updates for different attacks on the Purchase dataset. Our results show that the total  aggregated model updates under  existing attacks have  small magnitudes while those under our  attack have much larger magnitudes, which is caused by the inconsistent malicious model updates in existing attacks vs. consistent malicious model updates in our attack. 










 §.§ Impact of FL Settings


We study the impact of different FL settings, including the fraction of fake clients, degree of non-IID, the number of local training epochs, and fraction of clients that are selected to participate in each training round, on the effectiveness of existing attacks and our attack.
Unless otherwise mentioned, we show experimental results on CIFAR-10 dataset for simplicity, but we have similar observations on other datasets. Moreover, when studying the impact of one parameter, we use the default settings for other parameters discussed in Section <ref>. 






Impact of the fraction of fake clients 
Fig. <ref> shows the testing error rate of the global model as the fraction of fake clients increases from 1% to 20%. 
First, we observe that   is more effective, i.e., the testing error rate increases, as more  fake clients are injected. This is because the aggregated model update in each training round is more likely to have the sign vector s when more fake clients send the malicious model updates with the sign vector s to the server. 
Second, using only 5% of fake clients, our  can already break all FL defenses except Multi-Krum and FLAME.
However, existing attacks cannot reach such effectiveness even if 20% of fake clients are injected, except MPAF under Norm Bound defense.  



Impact of the degree of non-IID 
In our experiments, we use a parameter q to control the degree of non-IID among the genuine clients' local training data, where 0.1 ≤ q ≤ 1 <cit.>.  
A larger q indicates a larger degree of non-IID, where q=0.1 means the genuine clients' local training data is IID and q=1 indicates highly non-IID (i.e., each class of data is distributed among a subset of clients). 
Fig. <ref> shows the testing error rate of the global model as q increases from 0.1 to 1 under different attacks and defenses. 
We observe that  consistently achieves high testing error rates (i.e., random guessing) across different settings of q under different defenses. However, it is harder for existing attacks to compromise the defenses when q is smaller, e.g., q=0.1. 
This is because, in such cases, the  genuine model updates are less diverse, leaving less room for existing attacks to craft malicious model updates. We note that existing attacks are substantially less effective than  under Multi-Krum and FLTrust even if q=1.0. 

Impact of the local training epochs 
Fig. <ref> shows the testing error rate of the global model under different defenses and attacks as the number of local training epochs ranges from 1 to 10.
We observe that our  can effectively break the defenses (i.e., reducing the global models to be nearly random guessing) in all considered scenarios. Moreover, our  consistently achieves testing error rates no smaller than existing attacks. We do not observe any particular pattern with respect to how the effectiveness of an existing attack varies as the number of local training epochs changes. For instance, MPAF achieves a larger testing error rate as the number of local training epochs increases under Multi-Krum, but its testing error rate decreases under defenses like TrMean and Norm Bound. 

Impact of the Participation Rate
In each training round, the server typically selects a fraction (called participation rate) of the clients to participate in training. We use participation rate of 0.1 by default in our experiments. 
Fig. <ref> shows the testing error rates under different defenses and attacks when the participation rate ranges from 0.01 to 1, where FashionMNIST dataset is used. 
Note that for these experiments, we use FashionMNIST instead of CIFAR-10 dataset, due to limited memory of our GPU server. Moreover, we do not have results for Multi-Krum in Fig. <ref>  when the participation rate is 1 due to the same reason on insufficient memory. 
For FLCert, the smallest participation rate is set to 0.03 instead of 0.01, because FLCert divides the clients into 10 groups,  and if a participation rate of 0.01 is used, only one client is selected to participate in each training  round. Consequently, attacks requiring genuine local models are not applicable.
We observe that  participation rate almost has no impact on the effectiveness of both existing attacks and our attack. 
Moreover, our attack consistently outperforms existing attacks across all considered participation rates.



 §.§ Impact of Different  Variants


 crafts a malicious model update in each training round as a product of a given sign vector and a dynamically set magnitude vector, which is further decomposed into a product of an unit magnitude vector and a scaling factor. The unit magnitude vector and scaling factor play important roles in a malicious model update. 
Thus, we study different variants for designing them. We show results on CIFAR-10 dataset for simplicity. 
Due to limited space, we defer the discussion on impact of the sign vector in Appendix <ref>.


Variants of the unit magnitude vector 
For the unit magnitude vector, we consider the following two variants.

    
  * Same magnitude: 
    In this variant, the attacker sets each dimension of the  unit magnitude vector v^t to the same value. In other words, we set each dimension to be 1/√(d), where d is the number of dimensions/parameters.

    
  * Adaptive magnitude: In this variant, the dimensions of the unit magnitude vector have different magnitudes and are dynamically set in each training round based on how the global model changes and the malicious model updates in the previous training round. Equation <ref> shows our  adaptive unit magnitude vector adopted by .  
    

 Fig. <ref> shows the testing error rates under different defenses when  uses the two variants to set the unit  magnitude vector, where the scaling factor is set using our adaptive method in Equation <ref>. Our results show that adaptive magnitude substantially outperforms same magnitude for some defenses (e.g., Multi-Krum and FLTrust) while the two variants achieve comparable effectiveness for other defenses. This is because  adaptive magnitude can target specific dimensions with larger effectiveness. 


Variants of the scaling factor
We also consider two variants of setting the scaling factor as follows:

    
  * Maximized scaling factor: This variant sets a very large scaling factor with a goal to maximize the magnitude of the aggregated model update.  We use 100,000 in our experiments.
    
    
  * Adaptive scaling factor: This variant, adopted by ,  leverages how the global model changes in the previous training round to set the scaling factor (i.e., Equation <ref>) to avoid that the malicious model updates are filtered out by the server. 


Fig. <ref> illustrates the testing error rates under different defenses when  uses the two variants of setting the scaling factor, where adaptive magnitude is used to set the unit magnitude vector. We observe that, under some defenses (e.g., Median, TrMean, and Norm Bound), the variant of maximized scaling factor achieves comparable  testing error rates with adaptive scaling factor. This is because, for dimension-wise aggregation rules like Median and TrMean, although the malicious model updates with very large magnitudes are filtered out, they still deviate the aggregated model update; and although Norm Bound normalizes the norms of malicious model updates, the normalized malicious model updates still maintain consistency across training rounds. However, for other defenses like Multi-Krum and FLDetector that filter out entire model updates, adaptive scaling factor substantially outperforms maximized scaling factor. This is because these defenses filter out the entire malicious model updates with very large magnitudes. In contrast,  adaptive scaling factor leverages how the global model changes in the past  to achieve a balance between attack effectiveness  and undetectability. Since  aims to be defense-agnostic, adaptive scaling factor is preferred. 




§ COUNTERMEASURES AND ADAPTIVE ATTACKS









We explore new defenses tailored to . Specifically, we study a new defense that normalizes the total aggregated model update since  aims to increase its magnitude.  We also explore new tailored defenses to detect fake clients based on how  crafts the malicious model updates. We show that the normalization-based defense has limited effectiveness at mitigating  and we can still adapt  to break the detection based defenses. 

In our tailored detection based defenses, the server extracts a feature for each client in each training round and then divides the clients into two clusters via fitting a two-component Gaussian Mixture Model (GMM) <cit.> for the features.  We detect a particular cluster (e.g., with the smaller mean value) as fake clients. After detecting fake clients, the server removes them and re-trains a global model using the remaining clients. Note that if the distance between the mean values of the two clusters is smaller than the intra-cluster standard deviations of both clusters, no clients are detected as fake and the training continues since the difference between the two clusters may simply be caused by randomness. 

We extract features for clients based on how  crafts malicious model updates. Depending on which information of the malicious model updates is used to extract features, we consider two variants of GMM, i.e., GMM-Sign and GMM-Magnitude, which extract features based on the magnitudes and signs of the model updates, respectively. 



 §.§ GMM-Sign

Feature GMM-Sign exploits the consistency of the malicious model updates to detect fake clients. Specifically, for a  client i in training round t,  we define the feature x_i^t  as the  number of flipped dimensions of its model updates in two consecutive training rounds averaged in the recent N training rounds. Formally, we have: 

    x_i^t = ∑_t^' = t-N+1^t∑_j=1^d𝕀(sign(g_i^t'[j]) ≠sign(g_i^t'-1[j])),

where d is the number of dimensions/parameters of a model update;  𝕀 is the indicator function that has a value of 1 if sign(g_i^t'[j]) ≠sign(g_i^t'-1[j]) and 0 otherwise; and g_i^t'[j] is the j-th dimension of g_i^t'. For , we have x_i^t = 0 for all fake clients. 
After dividing the clients into two clusters, the cluster with a smaller mean value is detected as fake clients.

Adaptive attack
In response to the defense, we adapt .  
Specifically, we  randomly flip α (called flipping rate) fraction of the signs in each fake client's malicious model update and adjust the magnitudes of the corresponding dimensions to a small random value. 
Formally, we set the d-th dimension of the sign vector s^t_i and magnitude vector k^t_i for fake client i in training round t  as follows: 

    s^t_i[j]    =  
    -s[j]   with probability α
    s[j]   with probability  1-α,   
    k^t_i[j]    =  ϵ   if s^t_i[j] = -s[j]
    k^t[j]   otherwise ,

where j∈[1,d], s is the sign vector picked randomly at the beginning of an attack, k^t is the magnitude vector for training round t crafted by , and ϵ is a small value. 

Experimental results Fig. <ref> shows the detection accuracy of GMM-Sign and the testing error rates as the flipping rate α varies when the server uses different aggregation rules, where the dataset is CIFAR-10 and ϵ=1e-6.  We observe that GMM-Sign can accurately detect the fake clients when α is small. However, the detection accuracy reduces to 0 and -Adapt still compromises the learnt global models when α is larger than a threshold, e.g., 0.08. 


 §.§ GMM-Magnitude



Feature  uses the same malicious model update (i.e., g_i^t =k^t ⊙s) on all fake clients in training round t. Therefore, the server can extract  features for the clients based on the similarity between their model updates.  Specifically, for each client i in training round t, we propose to calculate a feature x_i^t as the sum of the average square distance between the client's model update  and its  closest m-1 model updates in the recent N training rounds. Formally, we have:  

    x_i^t = ∑_t^' = t-N+1^t∑_j ∈ℕ_i^t'g_i^t^'-g_j^t^'^2/m-1,

where ℕ_i^t' is the set of m-1 clients whose model updates are the closest to that of client i in round t' and m is the number of fake clients. Note that we give advantages to the defense by assuming that the number of fake clients is known to the server. In a round, the server divides the clients into two clusters via GMM based on the features. The cluster with a smaller mean value is detected as fake clients. The mean value is zero for  since the malicious model updates are the same for the fake clients in a round. 


Adaptive attack In response to such a defense, we can further adapt our  by introducing random noise to the magnitude vector k^t. Specifically, we can add a normalized noise to the magnitude vector for each fake client i as follows: 

    k^t_i = k^t 
     + γ· ||k^t||/||ϵ_i|| ·ϵ_i,

where  k^t is the magnitude vector crafted by , k^t_i is the magnitude vector for fake client i crafted by -Adapt, ϵ_i ∼𝒩(0, 𝐈) is the random noise sampled from the standard Gaussian distribution, ||k^t||/||ϵ_i|| normalizes the random noise to have the same magnitude as k^t, and γ is a scaling factor for the random noise. The noise makes the malicious model updates of the fake clients less similar to each other and thus evade detection. 


Experimental results Fig. <ref> shows the detection accuracy of GMM-Magnitude and the testing error rates of the global models as the noise scaling factor γ varies when the server uses different aggregation rules, where the dataset is CIFAR-10 and N=20.  
We observe that when γ is small, i.e., a small amount of noise is added to each malicious model update, GMM-Magnitude can accurately detect the fake clients and thus our -Adapt is not effective. However,  when γ is larger than a threshold (e.g., 1.2), GMM-Magnitude fails to detect the fake clients and -Adapt still makes the learnt global models nearly random guessing (i.e., testing error rates are around 90%). This is because fake clients and genuine clients have distinguishable features when  γ is small while their features are indistinguishable when γ is large. -Adapt still substantially increases the testing error rates when γ is large because the malicious model updates still have consistent sign vectors. 



 §.§ Normalization-based Defense

  aims to increase the magnitude of the total aggregated model update, i.e., w^T - w^0. Therefore, one tailored defense is to normalize the total aggregated model update. Specifically, the server can normalize w^T-w^0 to have a predefined ℓ_2 norm b and add the normalized total aggregated model update to the initial global model w^0 to get a new global model, i.e., w^T ←w^0 + bw^T-w^0/w^T-w^0. Fig. <ref> in Appendix shows the testing error rate of the final global model under  as the predefined norm b varies. Our results show that  the testing error rate can be decreased by this defense but is still high. This is because the update direction of the total aggregated model update is randomly picked by the attacker, which normalization cannot change. 



§ DISCUSSION AND LIMITATIONS






Using synthetic data In our previous experiments, when applying  attacks that require genuine clients' local models to craft malicious model updates, we assume the attacker has access to the local models on all genuine clients. This assumption is strong and not realistic. Therefore, we also explore that an attacker uses the global models to reconstruct synthetic data, train local models using the synthetic data, and then craft malicious model updates with them.  


Reconstructing training data from a model is known as model inversion <cit.>. 
 Previous work <cit.> shows that the global-model trajectory in multiple training rounds (especially early ones) of FL can be used to reconstruct synthetic data that is similar to the clients' genuine local training data. 
We follow the setting and generation strategy in <cit.>, which is the state-of-the-art reconstruction method in FL, to generate synthetic data for MNIST and FashionMNIST. We use  MNIST and FashionMNIST because they are simpler datasets and easier to reconstruct, giving advantages to these attacks. Specifically, we assume no attacks in the first 40 rounds and use the corresponding clean global models to reconstruct  synthetic data. Table <ref> shows the testing error rates of the classifiers trained on the synthetic data and evaluated on the genuine testing datasets of MNIST and FashionMNIST. The results show that the classifiers have decent testing error rates and thus the synthetic data mimic the genuine data well, which is consistent with prior work <cit.>.









We then use the synthetic data on the fake clients to perform model poisoning attacks in the remaining training rounds. Since each genuine client has 50 local training examples, we sample 50 synthetic training examples for each fake client. In each training round, the fake clients selected to participate in training train local models using their synthetic local training examples; and then we craft malicious model updates based on the local models for existing attacks.  
Table <ref> shows the testing error rates of the final global models under different attacks and defenses. We observe that an existing attack achieves higher testing error rates when having access to local models on the genuine clients in most cases, compared to using local models trained on synthetic data.   does not require genuine local models nor synthetic data but substantially outperforms these attacks. 


Cross-silo FL 
 primarily targets cross-device FL <cit.>, in which the clients are end-user devices such as smartphones. An attacker can inject fake clients into a cross-device FL system due to the open nature of the system. In particular, any client including fake ones can participate in the FL system. We acknowledge that it is harder to apply both  and existing  attacks to  cross-silo FL. Specifically, the clients are often a small number of verified institutions such as banks and hospitals in cross-silo FL. These institutions often go through certain verification processes before jointly training a model using FL. As a result, it is harder for an attacker to compromise genuine clients/institutions or inject fake clients/institutions in such a cross-silo FL system. We believe it is an interesting future work to identify a realistic threat model for cross-silo FL that considers the unique challenges and constraints.





§ CONCLUSION

In this work, we uncover the fundamental self-cancellation issue of existing model poisoning attacks to FL. Based on our discovery, we propose , a new model poisoning attack to FL. 
We show that, by maintaining multi-round consistency, an attacker can substantially increase the testing error rate of the learnt global model, even if the attacker does not have access to the genuine local training data, genuine local models, or deployed defense.  
 highlights the severe vulnerability of FL and the urgency for developing new defense mechanisms.




ACM-Reference-Format






§ ADDITIONAL EXPERIMENTAL RESULTS









 §.§ Analyzing the Effectiveness of PoisonedFL
















Why  breaks FL defenses 
In this section, we provide a detailed analysis of why  breaks existing defenses as follows:


    
  * Median and TrMean Even though the malicious model updates are not necessarily selected, their consistent nature over multiple training rounds biases the aggregation of each dimension towards the designated random update direction. 
    
    
  * Norm Bound Even if our attack effect is weakened in each training round, its cumulative effect over multiple training rounds remains pronounced with a significant magnitude for the total aggregated model update. 

    
  * Multi-Krum and FLAME For filtering-involved defenses based on Euclidean distance or cosine similarity, such as Multi-Krum and FLAME,  maintains effectiveness because the malicious model updates are not entirely filtered out in many training rounds due to dynamic magnitude adjustment. For instance, Table <ref> shows that a large fraction of all malicious model updates are not filtered out  by  FLAME and Multi-Krum with non-zero false negative rates.

    
  * FLTrust Table <ref> shows the normalized trust score of a malicious/benign model update of FLTrust across rounds. The normalized trust score is calculated as TS_i/∑TS, where TS_i is the trust score for client i, ∑TS is the summation of trust scores for all sampled clients in a round. The result shows that malicious model updates are not filtered out in multiple rounds (with a large normalized trust score), thereby the attack accumulates to corrupt the model. 
    
  * FLDetector The detection mechanism  FLDetector relies on inconsistent model updates of malicious clients. 
    Since the malicious model updates are consistent in , FLDetector falsely classifies all malicious clients as benign across all datasets.
    
  * FLCert FLCert is vulnerable to  since the global models for ensembling are learnt by existing aggregation rules, which are vulnerable to . 













 §.§ Impact of the Sign Vector




We further analyze the impact of the random sign vectors.
We experiment with PoisonedFL on CIFAR-10 with 6 random seeds (1-6) for various random sign vectors and defenses including TrMean, FLTrust, and FLAME, as shown in Table <ref>. The result aligns with our intuition that these random sign vectors do not affect the attack performance and PoisonedFL can successfully break the defenses with these 6 different random sign vectors. The reason is that in a continuous attack with the same random direction, the model inevitably updates substantially in a random direction in an uncontrolled way, and therefore leads to significantly degraded accuracy.  



§ ADDITIONAL DETAILS OF EXPERIMENTAL SETUP




 §.§ Datasets



    
    We use the following five datasets from different domains.
    
    MNIST <cit.> 
    MNIST is a 10-class handwritten digits classification dataset.
    It comprises 60,000 training examples and 10,000 testing examples. 
    In FL, the training data is typically not independently and identically distributed (non-IID) among clients. 
    Following <cit.>, we distribute the training examples among clients based on a non-IID degree of q = 0.5 by default.
    We train a convolutional neural network (CNN) for MNIST dataset. The architecture of CNN is shown in Table <ref> in Appendix.

    
    
    FashionMNIST <cit.> 
    FashionMNIST is a benchmark dataset of Zalando's article images, which contains 60,000 training examples and 10,000 testing examples. 
    For FashionMNIST, the degree of non-IID is also set to q=0.5 by default.
    We use the same CNN architecture for FashionMNIST dataset as employed in MNIST dataset.  

    Purchase <cit.> 
    Purchase is a 100-class customer purchase style prediction dataset. 
    Each input consists of 600 binary features.
    Following <cit.>, we split the total 197,324 examples into 180,000 training examples  and 17,324 testing examples. 
    Following <cit.>, we evenly distribute the training examples to clients and train a fully connected neural network.

     CIFAR-10 <cit.> 
    CIFAR-10 is 10-class color image classification dataset. 
    It comprises 50,000 training examples and 10,000 testing examples. 
    To consider a different degree of non-IID, we set q = 0.2 for CIFAR-10 dataset. 
    We train a CNN for CIFAR-10, whose  architecture  is shown in Table <ref>.

    FEMNIST <cit.> 
    FEMNIST is a  62-class classification dataset. 
     The dataset is already distributed among 3,550 clients with a total of 805,263 examples.
     We randomly sample 1,200 clients.
     We train a CNN for FEMNIST, whose architecture is shown in Table <ref>. 




 §.§ FL Settings


We primarily establish the FL setting in a cross-device scenario, which is more feasible for model poisoning attacks, as discussed in Section <ref>. Therefore, we assume 1,200 genuine clients and 20% fake clients.
The learning rates for the five datasets are selected within the range of 0.01 to 0.1 for optimal training effectiveness. Specifically, the learning rates for MNIST, FashionMNIST, and FEMNIST are set to 0.01, while CIFAR-10 is set to 0.03, and Purchase is set to 0.1.
The number of training rounds is configured to ensure full convergence under various scenarios: 6,000 for MNIST, FashionMNIST, and FEMNIST, and 10,000 for CIFAR-10 and FEMNIST.
The batch size is set to 32 for MNIST, FashionMNIST, CIFAR-10, and FEMNIST, and to 128 for Purchase, based on the number of local data samples.



 §.§ Defenses


We evaluate eight state-of-the-art defenses, including six Byzantine-robust aggregation rules (Multi-Krum <cit.>, Median <cit.>, Trimmed Mean <cit.>, Norm Bound <cit.>, FLTrust <cit.>, and FLAME <cit.>), one provably robust defense (FLCert <cit.>) and one malicious clients detection method (FLDetector <cit.>). We also consider the non-robust FedAvg <cit.> as a baseline. 




Multi-Krum <cit.>  
Multi-Krum  uses an iterative method to select a subset of clients' model updates. In each step, it selects the model update that has the smallest sum of Euclidean distance to its n-2 neighbors, where n is the number of genuine clients. 
This selection process continues until n clients are chosen. Finally, the server computes the average of the n selected model updates as the aggregated model update.



Median <cit.> 
In Median aggregation rule, the server computes the coordinate-wise median of the clients' model updates as the aggregated model update.


 Trimmed Mean (TrMean) <cit.> 
For each dimension, the server first removes the largest m values and smallest m values, and then computes the average of the remaining n-m values, where m is the number of  fake clients. 

Norm Bound <cit.>  
In Norm Bound aggregation rule, the server first clips each client's model update to have a predetermined norm and then computes the average of the clipped model updates as the aggregated model update.
We follow prior work <cit.> to set the predetermined norm  as the average norm of the genuine model updates in each training round.



FLTrust <cit.>
FLTrust assumes that the server has a small, clean  dataset to bootstrap trust. We assume the server's dataset includes training examples selected uniformly at random.  
The size of the server's dataset is  100 for MNIST, FashionMNIST, and CIFAR-10,  and 200 for  Purchase and FEMNIST.


 FLAME <cit.>  In each training round, FLAME considers the cosine similarity between clients' local models (i.e., a client's model update + current global model) to divide clients into clusters and filter out the clusters containing potentially malicious model updates.

 
FLCert <cit.> 
 FLCert divides clients into multiple groups and trains a global model for each group using any existing FL algorithm (we use Median <cit.>).   Given a testing input, FLCert predicts its label based on the majority vote among these global models. We divide the clients into 10 disjoint groups uniformly at random in our experiments. 




FLDetector <cit.>  
 FLDetector aims to detect malicious clients during training. 
 We first apply FLDetector with the full participation of clients for detection for 300 communication rounds. 
 Following <cit.>, if some clients are detected as malicious, we remove them from the system.
 Then we re-train a global model from scratch based on the remaining clients using Median <cit.> aggregation rule with the default setting.














 



 §.§ Compared Attacks


We compare our   with seven attacks, including five attacks (Fang <cit.>, Opt. Fang <cit.>, LIE <cit.>, Min-Max <cit.>, and Min-Sum <cit.>) that require genuine clients' information and two attacks (Random <cit.> and MPAF <cit.>) that do not require such information. 
Note that when applying the first category of attacks to craft the malicious model updates on fake clients, we assume that the attacker has access to the model updates of all genuine clients and the aggregation rule used by the server. In Section <ref>, we also study the scenarios where an attacker uses the global models to reconstruct synthetic data on the fake  clients and uses the local models trained on them to perform attacks. 
We do not make these assumptions for  Random, MPAF, and our . Therefore, we give advantages to the first category of attacks. 


    
Fang <cit.> 
In this attack, the attacker crafts malicious model updates for fake clients such that the aggregated model update after attack deviates substantially from the before-attack one. Fang has different versions for different aggregation rules. 
We apply their Krum attack for Multi-Krum defense and TrMean attack for others.

 
 
Opt. Fang <cit.> 
Following Fang, this attack also aims to maximize the deviation between the after-attack aggregated model update and before-attack one, but it uses a different way to solve the malicious model update. Specifically, a malicious model update is a variant of the average of all genuine clients' model updates.
 
 

A little is enough (LIE) <cit.> 
In LIE attack, fake clients craft their malicious model updates by adding a small amount of noise to the average of genuine model updates.
Specifically, for each dimension j, LIE calculates the mean μ_j and standard deviation σ_j among the genuine model updates, and the dimension j of the malicious model update is set as μ_j + 0.74 ·σ_j.





    Min-Max <cit.> 
In this attack, a fake client crafts its malicious model update such that its distance to any genuine model update is no larger than the maximum distance between any two genuine model updates.


    Min-Sum <cit.> 
Each fake client crafts its malicious model update such that the sum of the distances between the malicious model update and all genuine model updates is not larger than the sum of distances between any genuine model update and other genuine model updates.

    
    Random <cit.> 
    In this attack, each fake client i ∈[n+1, n+m] sends a scaled random vector g_i^t = λ·ϵ to the server, where ϵ∼𝒩(0, 𝐈) and λ is a scaling factor. 
    Following prior work <cit.>, we set λ=1e6 in our experiments.



        MPAF <cit.>
    In MPAF attack, during training round t, fake client i ∈[n+1, n+m] crafts its malicious model update as g_i^t=λ· (w^'-w^t-1), where w^' is an attacker-chosen random model, w^t-1 is the global model in the previous training round t-1, and λ is a scaling factor. Following prior work <cit.>, we set λ=1e6 in our experiments.






