The Best Defense Is a Good Offense: Adversarial Attacks to Avoid Modulation Detection
Muhammad Zaid Hameed Andras Gyorgy Deniz Gunduz







The Best Defense Is a Good Offense: Adversarial Attacks to Avoid Modulation Detection


This work was presented in part at the 7th IEEE Global Conference on Signal and Information Processing (GlobalSIP 2019) <cit.>.
This work was supported by an Imperial College London President's PhD Scholarship, and by the European Research Council (ERC) through the Starting Grant BEACON (No. 677854).

    Muhammad Zaid Hameed^1,  András György^2, and Deniz Gündüz^1


^1Department of Electrical and Electronic Engineering,
Imperial College London, UK 

^2DeepMind, London, UK 

Email: muhammad.hameed13@imperial.ac.uk, agyorgy@google.com, d.gunduz@imperial.ac.uk

    
======================================================================================================================================================================================================================================================================================================================================================================================================




We consider a communication scenario, in which an intruder
tries to determine the modulation scheme of the intercepted signal. Our aim is to minimize the accuracy of the intruder, while guaranteeing that the intended receiver can still recover the underlying message with the highest reliability. This is achieved by perturbing channel input symbols at the encoder, similarly to adversarial attacks against
classifiers in machine learning. In
image classification, the perturbation is limited to be imperceptible to a human observer, while in our case the perturbation is constrained so that the message can still be reliably decoded by the legitimate receiver, which is oblivious to the perturbation. Simulation results demonstrate the viability of our approach to make wireless communication secure against
state-of-the-art intruders (using deep learning or decision trees) with minimal sacrifice in the communication performance. On the other hand, we also demonstrate that using diverse training data and curriculum learning can significantly boost the accuracy of the intruder.



secure communication, deep learning, adversarial attacks, modulation classification




§ INTRODUCTION


Securing wireless communication links is as essential as increasing their efficiency and reliability, for military, commercial, as well as consumer communication systems. The standard approach to securing communications is to encrypt the transmitted data. However, encryption may not always provide full security (e.g., in case of side-channel attacks), or strong encryption may not be available due to complexity limitations (e.g., for IoT devices). To further improve security, encryption can be complemented with other techniques, preventing the adversary from even recovering the encrypted bits.

As outlined in <cit.>, an adversary implements its attacks on a wireless communication link in four steps: 1) tunes into the frequency of the transmitted signal; 2) detects whether there is signal or not; 3) intercepts the signal by extracting its features; and 4) demodulates the signal by exploiting the extracted features, and obtains a binary stream of data. Preventing any of these steps can strengthen the security of the communication link. While encryption focuses on protecting the demodulated bit stream, physical layer security <cit.> targets the fourth step by minimizing the mutual information available to the intruder. Recently, there has also been significant interest in preventing the second step through covert communications <cit.>. In this work, we instead focus on the third step, and aim at preventing the adversary from detecting the modulation scheme used for communications.

Modulation detection is the step between signal detection and demodulation in communication systems, and thus plays an important role in data transmission, as well as in detection and jamming of unwanted signals in military communications and other sensitive applications  <cit.>.
Recently, deep learning techniques have led to significant progress in modulation-detection accuracy:  methods based on convolutional and other deep neural networks can detect the modulation scheme directly from raw time-domain samples <cit.>, surpassing the accuracy of conventional modulation detectors based on likelihood function or feature-based representations (see <cit.> for a survey of these approaches).



Our aim in this paper is to prevent an intruder that employs a state-of-the-art modulation detector from successfully identifying the modulation scheme being used. The rationale behind this is that if the intruder is unable to identify the modulation scheme, it is unlikely to be able to decode the underlying information or employ modulation-dependent jamming techniques to prevent communication. To achieve this goal, we introduce modifications to the transmitted signal. The main challenge here is to guarantee that the intended receiver of the (modified) transmitted signal can continue to receive the underlying message reliably, while preventing the intruder from detecting the modulation scheme being used. Otherwise, reducing the accuracy of the modulation-detecting intruder would be trivial by sacrificing the performance of the intended receiver. We assume that the intended receiver is oblivious to the modifications employed by the transmitter to confuse the intruder, and, therefore, the goal of the transmitter is to introduce as small modifications to the transmitted signal as possible that are sufficient to fool the intruder but not larger than the error correction capabilities of the intended receiver.

Introducing small variations into the modulation scheme that can fool 
an intruder is similar to adversarial attacks on classifiers, in particular, deep neural networks (DNNs) <cit.>. In the literature, adversarial attacks are mostly considered in the area of image classification, where they pose security risks  by exposing the vulnerabilities of classifiers against very small changes in the input that are imperceptible to humans but lead to incorrect decisions. In contrast, we exploit the same approach here to defend a communication link against an intruder that employs DNNs or other standard classification methods for interception.

In <cit.>, an adversarial attack for a deep-learning-based modulation classifier has been proposed where the adversary assumes the availability of noisy symbols received  at the modulation classifier for generating the adversarial attack, which makes it impractical and limited in scope. A similar method has been proposed  recently in <cit.>, where modifications are employed by the transmitter to evade a DNN-based jammer, and the receiver uses another DNN (an autoencoder) to preprocess the received signal and filter out the modifications. However,
no analysis has been provided on the impact of this method
on the bit error rate (BER) of the received signal. In contrast to <cit.>, we do not limit our approach to a DNN-based jammer and consider a
receiver that is completely oblivious to the modifications in the transmitter.

We also consider the impact of the
defensive perturbations on the BER at the legitimate receiver.
The results of <cit.> has been extended in <cit.> to the detection of wireless communication protocols, and targeted adversarial attacks are also considered to generate the perturbations.[Targeted adversarial attacks <cit.> aim to modify the data so that the attacked classifier predicts an incorrect class selected by the attacker.]

A number of concurrent works have also appeared in the literature (parallel or following the original publication of our preprint at arXiv <cit.> and the conference version of our paper <cit.>). Most similar to ours is <cit.>, which proposes modifications in the transmitted signal using an adversarial residual network at the transmitter to evade the modulation detector at an intruder while the legitimate receiver is able to decode the signal with small bit error rate. Compared to this paper, we use different adversarial attack techniques, propose different ways of improving the modulation-detection accuracy of the intruder, and analyze the trade-off between the code rate and the BER for defensive perturbations and an improved intruder.

Adversarial perturbations have also been applied to attack a legitimate receiver in <cit.>. In these works the input signal of the receiver is perturbed by an over-the-air attack to make the modulation classifier of the legitimate receiver fail (cf. in our case the transmitter changes the signal to fool the intruder). In <cit.>, the performance of such attacks is evaluated in terms of the BER at the receiver under the impractical assumption that the attacker has full knowledge of the signal at the receiver. The same scenario has been considered in <cit.>, and attack methods of various strength have been devised under more realistic assumptions about the capabilities of the attacker, in particular about its information on the signal received by the modulation classifier (fully known vs. its distribution being estimated based on samples available to the attacker) and on the channel noise from the attacker to the receiver (knowing the exact realization or just the noise distribution). While these attack methods share the underlying idea with our defensive perturbations, they face a much easier problem, as the attacks are not constrained by ensuring a low BER at a distinct receiver.

While we consider adversarial attack methods that affect the behavior of trained classifiers (i.e., the modulation classifier of the intruder in our case) by perturbing their input data (these attacks are known as test-time or evasion attacks), another class of adversarial machine learning algorithms, called poisoning attacks, aim to compromise the training procedure of classifiers and other machine learning models by modifying their training data <cit.>. Poisoning attacks have been used in launching and avoiding jamming attacks in wireless communication <cit.>; however, since these methods address the training of the machine learning models employed by the jammer and the transmitter, they are orthogonal to our developments.


In summary, our main contributions are as follows:


    
  * We propose a novel defense mechanism that modifies the channel input symbols at the transmitter in order to reduce the modulation-classification accuracy at the intruder while maintaining a low BER at the legitimate receiver.
    
  * We provide a thorough experimental evaluation of the effect of these modifications on the BER of different modulation schemes.
    
  * We demonstrate that by using training data obtained from different SNR values and employing curriculum learning, an intruder can learn a classifier that is much more robust against both the channel noise and the defensive perturbations, improving upon the state of the art in our experiments when no defense mechanism is applied.
    
  * We show that by reducing the communication rate, the BER at the legitimate receiver can be reduced  while the intruder is limited to achieve the same or worse modulation-classification accuracy.


The rest of the paper is organized as follows: The system model is described in Section <ref>, followed by the description of our novel modulation perturbation methods in Section <ref>. Experimental results are presented in Section <ref>, while conclusions are drawn and future work is discussed in Section <ref>.



§ SYSTEM MODEL



Consider a transmitter that maps a binary input sequence ∈{0, 1}^m into a sequence of n complex channel input symbols, ∈^n, employing forward error correction coding. The input data is first encoded by the channel encoder, and then modulated for transmission.
Formally, the modulated signal  is obtained as =M_s(), where s ∈ is the employed modulation scheme with  denoting the finite set of available modulation schemes, and for any s, M_s:{0,1}^m →^n denotes the whole encoder function with modulation s. We assume that M_s satisfies the power constraint (1/n) _2^2 ≤ 1 for any input sequence . After encoding, signal  is sent over a noisy channel, assumed to be an additive white Gaussian noise (AWGN) channel for simplicity: baseband signals _1 and _2, received by the intended receiver and the intruder, respectively, are given by

    _i = M_s() + _i =  + _i,  i=1,2,

where _1,_2 ∈^n are independent channel noise (also independent of ) with independent zero-mean complex Gaussian components with variance σ_1^2 and σ_2^2, respectively.

The intended receiver, upon receiving the sequence of noisy channel symbols 𝐲_1, demodulates the received signal, and decodes the underlying message bits with the goal of minimizing the (expected) BER 𝔼[𝕖(𝐰, 𝐲_1)], where

    e(𝐰,_1) ≜1/m∑_i=1^m w_i≠ŵ_i,

ŵ is the decoded bit sequence from _1, and the expectation is over the uniformly random input bit sequence 𝐰 and the noise sequence 𝐳_1.[For any event E, E=1 if E holds, and 0 otherwise. Furthermore, for any real or complex vector 𝐯, v_i denotes its ith coordinate.]

The intruder aims to determine the modulation scheme employed by the transmitter based on its received noisy channel output _2. The transmitter, on the other hand, wants to communicate without its modulation scheme being correctly detected by the intruder, while keeping the BER at an acceptable level.

Formally, the aim of the intruder is to determine, for any sequence of channel output symbols _2 ∈^n, the modulation method used by the transmitter. This leads to a classification problem where the label s ∈ is the employed modulation scheme, and the input to the classifier is the received channel sequence _2 ∈^n. We consider the case in which the intruder implements a score-based classifier, and assigns to _2 the label ŝ=_s' ∈ f_(_2,s'), where f_: ^n ×→ is a score function  parametrized by ∈^d, which assigns a score (pseudo-likelihood) to each possible class s' ∈ for every _2, and finally selects the class with the largest score. With a slight abuse of notation, we denote the resulting class label by ŝ = f_(_2).
The goal of the intruder is to maximize the probability (s = ŝ) of correctly detecting the modulation scheme, which we will also refer to as the success probability of the intruder.[Here we assume an underlying probabilistic model about how the the bit sequence w and modulation scheme is selected.]
For state-of-the-art modulation detection schemes <cit.>, f_ is a convolutional neural network classifier,  is the vector of the weights of the neural network, while the f_(_2,s') are the so-called logit values for the class labels s' ∈.

The performance of both the intended receiver, measured by the BER, and the intruder, measured by the detection accuracy, depend on the signal-to-noise ratio (SNR) of the corresponding channels, 1/σ_1^2 and 1/σ_2^2, respectively. We assume that these SNR values are known by the legitimate receiver and the intruder, which can employ a specific f_ for each SNR value. We will also assume that the intruder has access to training data at the SNR value 1/σ_2^2 to train f_. This can be done offline as the intruder can generate as much training data as required at a specific SNR value.




§ MODULATION PERTURBATION TO AVOID DETECTION


In this paper we intend to modify the encoding processes M_s such that, given a modulation scheme s ∈, the new encoding method M'_s ensures that the intruder's success probability gets smaller, while the BER of the receiver (using the same decoding procedure for M_s) does not increase substantially. Our solution is motivated by adversarial attacks for image classification, where it is possible to modify images such that the modification is imperceptible to a human observer, but it makes state-of-the-art image classifiers to err <cit.>. Adversarial examples are particularly successful in fooling high-dimensional DNN classifiers. Applying the same idea to our problem, we aim to find defensive modulation schemes M'_s such that M'_s() ≈ M_s(), but the intruder misclassifies the new received signal '_2=M'_s()+_2 with higher probability.



 §.§ Adversarial attack in an idealized scenario


Following directly the idea of adversarial attacks on image classifiers <cit.>, an idealized yet impractical adversarial attack mechanism is proposed in <cit.> which modifies a correctly classified channel output sequence _2 (i.e., for which s=f_(_2)) with a perturbation ∈^n such that f_(_2+) ≠ f_(_2), the true label, while imposing the restriction _2 ≤ϵ for some small positive constant ϵ.
Thus, to mask the modulation scheme, the goal is to find, for each correctly classified _2 separately, a perturbation  that maximizes the zero-one loss:

    maximize  f_(_2+) ≠ s such that  _2 ≤ϵ ,

where s=f_(_2) is the true modulation label.

If the maximum is 1, such a  results in a successful adversarial perturbation and a successful adversarial example _2+ (i.e., one for which the intruder makes a mistake).
This approach, however, has two limitations. First of all, as opposed to image classifiers, we are not concerned with the visual similarity of the perturbed signal _2 + to the original one, _2. The reason for bounding the perturbation  is instead to guarantee that the BER at the intended receiver is still limited. Moreover, in practice we do not have access to _2, as it does not only depend on , but also on the channel noise _2, which is not available at the transmitter. Therefore, the above mechanism, analyzed in <cit.>, is an oracle scheme working under some idealized assumptions, and we use it only as a baseline.

It remains to give an algorithm that finds an adversarial perturbation  solving problem (<ref>). However, we note that the target function f_(_2+) ≠ f_(_2) is binary, and so no gradient-based search is directly possible. To alleviate this, usually a surrogate loss function L(,_2,s) to the zero-one loss is used (which is often also used in training the classifier f_), which is amenable to gradient-based (first-order) optimization. For classification problems, a standard choice is the cross-entropy loss defined as L(,_2,s) = -log(1+e^-f_(_2,s)), and one can search for adversarial perturbations by solving

    maximize   L(,_2+,s)
             such that  _2 ≤ϵ.




Different methods are used in the literature to solve (<ref>) approximately <cit.>.
In this paper we use the state-of-the-art projected (normalized) gradient descent (PGD) attack <cit.> to generate adversarial examples, which is an iterative method: starting from ^0=_2, at each iteration t it calculates

    ^t =  Π_ℬ_ϵ(_2)( ^t-1 + β sign(∇_ L(, ^t-1,s)) ),

where β>0 denotes the step size, sign denotes the sign operation, and Π_ℬ_ϵ(_2) denotes the Euclidean projection operator to the L_2-ball ℬ_ϵ(_2) of radius ϵ centered at _2, while ∇ denotes the gradient. The attack is typically run for a specified number of steps, which depends on the computational resources; in practice ^t is more likely to be a successful adversarial example for larger values of t. We will refer to this idealized modulation scheme as the Oracle Scheme ().

Note that this formulation assumes that we have access to the logit function f_ of the intruder; these methods are called white-box attacks. If f_ is not known, one can create adversarial examples against another classifier f_', and hope that it will also work against the targeted model f_. Such methods are called black-box attacks, and are surprisingly successful against image classifiers <cit.>. We will also consider black-box attacks against intruders in our experimental evaluations.




 §.§ Adversarial attack through channel input modification


As mentioned before, the scheme is infeasible in practice as the transmitter can only modify the channel input =M_s() but not _2 directly. Thus, the new modulation scheme is defined as


    M'_s()=α (M_s()+),


where we will consider different choices for ∈^n, and the multiplier α = √(n)/M_s()+_2 is used to ensure that the new channel input = M'_s() satisfies the average power constraint (1/n) _2^2 ≤ 1. The signals received at the receiver and at the intruder are _1 =  + _1 and _2 =  + _2, respectively.
The difficulty in this scenario is that the effect of any carefully designed perturbation  may be (and, in fact, is in practice) at least partially masked by the channel noise. Furthermore, since now the perturbed signal is transmitted at the actual SNR of the channel, the effective SNR of the system is decreased, as the transmitted signal already includes the perturbation , which can be treated as noise from the intended receiver's point of view.

Our first and simplest method to find a perturbation   disregards the effects of the channel noise and the resulting BER at the receiver.



  §.§.§ Perturbation-based Defensive Modulation Scheme (PDMS)
 In this method, called the , we aim to solve the optimization problem (<ref>) with  in place of _2, via (<ref>) initialized at ^0= and with projection to ℬ_ϵ() (for a specified number of iterations t and perturbation size ϵ).



 §.§ BER-aware adversarial attack

Next, we consider methods that also take into account the BER, e(_1,) at the receiver (see Eq. <ref>): that is, instead of enforcing the perturbation  to be small and hoping for only a slight increase in the BER, we optimize also for the latter. There is an inherent trade-off between these two targets: a larger  results in a bigger reduction in the detection accuracy of the intruder, but will also increase the BER at the receiver. We consider two methods to handle this trade-off:

In the first one, called BER-Aware Defensive Modulation Scheme (BDMS);
we consider a (signed) linear combination of our two target functions in order to balance the above two effects,

    L_λ(,,s,_1,_2)= L(,+_2,) - λ e(+_1,)

for some λ>0, where _i=+_i, i=1,2,
and aim to find a perturbation  or, equivalently, a modulated signal =+ that maximizes the expectation

    𝔼__1,_2 [L_λ(,,s,_1,_2)]

with respect to the channel noise _1,_2.
Here we can use stochastic gradient ascent[However, similarly to the literature on adversarial attack methods, we often call these methods gradient descent instead of ascent.] to compute an approximate local optimum, but in practice we find that enforcing  to be small during iterations improves the performance; hence, we use a stochastic version of PGD optimization (<ref>): starting at ^0=, our candidate for  is iteratively updated as

    ^t=Π_ℬ_ϵ()(^t-1 + β·sign ( ∇_L(,^t-1,s,^t_1,^t_2)) ),

where ^t_i are independent copies of _i, respectively, for i=1,2, and t=1,2,…. Although 𝔼__1 [e(+_1,)] is differentiable, e(,) for a given fixed value of  is not (since it takes values from the finite set {0,1/n,…,1}). Similarly to <cit.>, we approximate the gradient of the expected error using simultaneous perturbation stochastic approximation (SPSA) <cit.> as

    ∇_  e(,)
    =1/K∑_k=1^K e(+η_k,)-e(-η_k,)/2η_k^⊤,

where _1,…,_K are random vectors selected independently and uniformly from {-1,1}^n.


In the alternative BER-Aware Orthogonal Defensive Modulation Scheme (BODMS),
instead of maximizing the combined target (<ref>), we try to maximize the cross-entropy loss L(,_2,s) while not increasing (substantially) the BER e(_1,). In order to do so, we maximize L(,_2,s) using stochastic PGD (again, in every step we choose independent noise realizations), but we restrict the steps in the directions where the BER does not change. Thus, in every step we update ^t-1 in a direction orthogonal to the gradient of the BER defined as

    ∇_o L(,^t-1+^t_2,s) 
       ≜∇_ L(,^t-1+^t_2,s) -
    ⟨∇_ L(,^t-1+^t_2,s), d_e⟩ d_e

where
d_e= ∇_ e(^t-1+^t_1,)/∇_ e(^t-1+^t_1,)_2 is the (approximate) gradient direction of the BER (computed, e.g., using SPSA as in Eq. <ref>).





§ EXPERIMENTAL EVALUATION


In this section, we test and compare the performance of the proposed methods through numerical simulations. We assume that the binary source data is generated independently and uniformly at random, and is encoded using a rate 2/3 convolutional code before modulation. Eight standard baseband modulation schemes are considered: , , 8, , , 4, 16, 64. A square-root raised cosine filter is used for pulse shaping of the modulated data with a filter span of 10,  roll-off factor of 0.25 and upsampling factor of 8 samples per symbol and the modulated data is sent over an AWGN channel with SNR varying between -20 dB and 20 dB. We consider identical SNRs during both the training of the intruder and at test time. After hard decision demodulation, the receiver uses Viterbi decoding to estimate the original source data.

We follow the setup of <cit.> for modulation detection: The intruder has to estimate the modulation scheme after receiving 128 complex I/Q (in-phase /quadrature) channel symbols;  this is because we assume that the modulation detection is only the first step for the intruder, which then uses this information for either trying to decode the message or to interfere with its transmission. Therefore, the modulation detection should be completed based on a short sequence of channel symbols. As the classifier, we first consider the deep convolutional neural network architecture of <cit.> for the intruder
which operates on the aforementioned 256-dimensional data.

For each modulation scheme, we generate data resulting in approximately 245000 I/Q channel symbols (note that for different modulation schemes this corresponds to different number of data bits), split into blocks of 128 I/Q symbols (n=128), as explained above. The last 300 blocks for each modulation scheme are reserved for testing the performance (tests are repeated 20 times), while we train a separate classifier for each SNR value based on the above data. As shown in  Fig. <ref> (see the curve with label ), for high SNR values the accuracy of the modulation classification is close to 90%. 
As expected, the classification accuracy degrades as the SNR decreases (as the noise masks the signal), but even at -10 dB, the intruder can achieve a 40% detection accuracy (as opposed to the 12.5% accuracy a completely random detector would achieve).



In the experiments, we compare this performance with

    
  * our three defensive modulation schemes, , , and , as well as as baseline;
    
  * adding uniform random noise of L_2-norm ϵ to a block, called random noise insertion (), which is then normalized for power constraints;
    
  * a black-box attack mechanism that does not use the classifier of the intruder, but calculates against a classifier that has the same architecture as that of the intruder's but is trained separately (assuming no channel noise); we call this the Blackbox ().


All the above schemes, except for , are implemented using the projected (normalized) gradient descent (PGD) <cit.> method from the CleverHans Library <cit.>, with 20 iterations, β=0.2 and ϵ=3. ϵ=3 results in significant reduction in modulation-classification accuracy without incurring too large BER at the intended receiver and has been determined by running experiments over different values of ϵ. uses the same ϵ. Note that a perturbation of this size accounts for about 7% of the total energy of a block (which is 128 due to our normalization to the energy constraint). serves as an upper bound on the achievable defensive performance given the parameters, while the role of is to analyze the effect of carefully crafted perturbations instead of selecting them randomly. explores the more practical situation where the exact classifier of the intruder is not known, but its training method and/or a similar classifier is available.



 §.§ Defensive modulation schemes with norm-bounded perturbations

We first consider defensive modulation schemes with a bound on the L_2 norm of the applied perturbation. Fig. <ref> shows the modulation-classification accuracy for several methods. It can be seen that adding random noise () helps very little compared to no defense at all (). The basic defense mechanism and its black-box version become effective from about -5 dB SNR, and; as expected; outperforms . For smaller SNR values the classification accuracy is relatively small (the channel noise already makes classification hard), and only the oracle defense gives noticeable improvement. As expected, the performance of gets closer to its lower bound, , as the SNR increases (note that the two methods coincide at the limit of infinite SNR).  The similar performance of and for medium SNR values shows a similar transferability of adversarial attacks in our situation as was observed in other machine learning problems, such as in image classification <cit.>,
although this effect deteriorates quickly as the SNR increases and becomes more effective. Observe that the classification accuracy of increases up to 0 dB SNR, when the channel noise during both the training phase and test phase is higher than the defensive perturbation and thus, channel noise is the main cause of the performance limitation of the intruder, while the accuracy decreases for higher SNR when the defensive perturbation is larger compared to the channel noise and the defense mechanisms start working.

Table <ref> shows the modulation-classification accuracy for the individual modulation schemes at channel SNR of 20 dB. It can be seen that a defensive perturbation of the same norm ϵ affects different modulation schemes differently, where and appear to be the most robust against defensive perturbations. Note that  16 and 64 are very difficult to classify even without any perturbations, which is in line with the observation made in <cit.>. Modulated signals without any perturbation and -modulated signals are presented in Fig. <ref>, which shows that, even after perturbation, retains the modulated signal constellation and the perturbed signals are still different from the output of any other modulation scheme. On the other hand, it becomes difficult to distinguish 16 and 64 signals.






The reduced classification accuracy of the intruder for and are countered by the increased BER at the receiver. To illustrate this effect, Fig. <ref> shows the BER for 8 and 64; the other modulation schemes, except for 16, show similar relative behavior to 8, but with the error dropping sharply for medium SNR values, with a few dB difference among different modulation schemes (up to about 5 dB for 8). On the other hand, the price of using any defense mechanism on 64 is severe, resulting in a significantly higher BER in the high SNR regime; 16 behaves similarly with somewhat smaller BER values. For the defensive scheme, we directly feed the perturbed signal to the decoder to calculate the BER, which is lower than the BER at the decoder when the and defensive schemes are employed for 8, while these BERs are essentially the same for 64 .

This negative effect on the BER can be suppressed if the perturbation size is decreased, which, at the same time, results in increased detection accuracy. This is shown in Fig. <ref> as a function of the signal-to-perturbation ratio SPR ≜ n/_2^2 (recall n=128, and SPR≈ 11.5dB corresponds to ϵ=3).
In every case, trades off increased BER for reduced detection accuracy compared to the case when no defense mechanism is applied. Also, increasing the number of iterations used in the defensive schemes to compute the perturbations has limited impact on modulation-classification accuracy and BER as the total perturbation is limited to have L_2-norm ϵ.




Fig. <ref> shows the trade-off between the average modulation-detection accuracy of the intruder and the BER for the individual modulation schemes for an intruder DNN trained at an SNR of 10 dB (i.e., the training samples are generated with this channel SNR) when the maximum perturbation norm ϵ of takes values in the range [1,6] (smaller ϵ values correspond to points with smaller BER and larger classification accuracy on each curve). It can be seen that an effective perturbation that results in a reduction in the modulation-classification accuracy also causes an increase in the BER. The trade-off between the two is different for different modulation schemes for the same perturbation constraint ϵ  (note that the reported classification accuracy is an average computed over all modulation schemes) . It can be seen that an increase in ϵ needed to reduce the average modulation-classification accuracy results in large BER for 16 and 64. Note that in our experiments , , and have zero error rate for this ϵ range, hence they are not included in the figure.







 §.§ BER-aware defense schemes


A more systematic way of improving the BER is to use our BER-aware modulation schemes and .
In the numerical experiments, due to the large computational overhead of calculating the SPSA gradient estimates in (<ref>) (with K=400), we only used 400 signal blocks to measure the test performance (instead of the 300×20=6000 blocks used previously).
Also, to keep the required computation feasible, in (<ref>) we used error rates calculated over 100 signal blocks (that is, over 12800 perturbed channel input symbols simultaneously). This approximation allowed us to run Viterbi decoding once for every hundred blocks, instead of running it from the beginning for every block, causing a substantial reduction in computational complexity. The approximate gradient of e computed this way was then used to calculate one step of the optimization (i.e., the next candidate perturbation) for each of the 100 blocks simultaneously. The drawback of this approximation is twofold: (i) instead of taking the gradient for a single perturbation, for each perturbation the error gradient is computed as an average coming from perturbing each of the 100 blocks simultaneously (this affects negatively the accuracy of the optimization); (ii) the applied method introduces delays in the transmission as it assumes that all signal blocks perturbed together are available at the transmitter at the same time (this gives some optimistic bias to the optimization compared to non-delayed real-time encoding).
Nevertheless, we believe that the negative effects are stronger here, and the performance of our modulation schemes (and ) could be improved if the BER of the individual signal blocks were used for gradient estimation in SPSA.

Fig. <ref> and Fig. <ref> show, respectively, the modulation-classification accuracy and the BER for and , also compared to , and , against a DNN-based intruder, which is trained with channel input symbols at specific SNR values. The performance of is presented for three different values of λ, namely 1, 10^3, 10^6. As before, the BER is shown for 8 and 64, as again 64 is the modulation scheme most affected by our perturbations, and except for 16 (which is similar to 64), and the error rate for the other modulation schemes is similar to (in fact smaller than) that of 8 and is very small under any defense mechanisms at high SNR values.

It can be seen that at high SNR (at least 12 dB), all defensive schemes achieve roughly the same classification accuracy, while and for large λ provide significant improvement in the BER (shown for 8 and 64). Note, however, that the errors are still significantly higher than for the standard 64 modulation with no perturbation.

For larger λ values, the BER of for 64 is smaller than or approximately the same as for , which adds uniform random noise of the same perturbation size, while it significantly outperforms in classification accuracy (for 8, both and achieve low BER, although it can be much smaller for ). Note that approaches the performance of with a large λ (10^3-10^6), without the need to tune the hyperparameter λ, and these methods provide a good compromise between the effectiveness of the defense and the increase in the BER.

In addition to DNN-based detectors at the intruder, we also examine defense against one of the best standard modulation detection schemes in the literature, a multi-class decision tree trained with expert features obtained from <cit.>.
Fig. <ref> shows the modulation-classification accuracy and the BER achieved by employing various defense mechanisms against this intruder. It can be seen that the BER achieved against the tree-based classifier is approximately the same as the one achieved against the DNN-based classifier with and , while the accuracy of the DNN-based classifier is consistently higher, except for some high SNR values, when they are approximately the same. This demonstrates that our observations and conclusions also apply to intruders employing other types of detection mechanisms.










 §.§ Robustness of the intruder's classifier


In the previous sections we assumed that the intruder knows the SNR of its received signals perfectly and trains its classifier for this SNR value. Although this may not be possible in practice due to estimation errors or variations in channel quality, assuming more accurate information at the intruder should allow us to design stronger defense mechanisms. In this subsection, we study the robustness of the intruder's detection network against errors in its SNR estimate; that is, we study its modulation-detection accuracy when it is trained for a specific channel SNR, but tested at different SNR values. We show in Fig. <ref> the results for three cases: (a) when no defense mechanism is applied (i.e., ); (b) when uniform noise is added (); and
(c) when our perturbation-based defense is applied. In each figure, we plot the detection accuracy with respect to the test channel SNR when the intruder is trained at five different SNR values. represents the case in which the test channel SNR matches the training SNR.
 


We can observe in Fig. <ref> that the intruder network trained at channel SNR -20 dB is unable to learn any effective classifier for higher SNR values. As the channel SNR at the time of training increases, its performance improves for a larger range of test SNR values as evident from the plots for SNR -10 dB and 0 dB, but, as one would expect, the accuracy achieved is below the peak accuracy values in the curve. On the other hand, networks trained with high SNR values of 10 dB and 20 dB achieve higher accuracy, close to peak accuracy values in the curve, but tend to breakdown when SNR goes below a certain value (2 dB and 6 dB for intruder networks trained at 10 dB and 20 dB, respectively). It is due to the fact the DNNs learn the classifier function (decision boundaries) from the training data, and for those trained at high channel SNR, signals with higher noise may lie across decision boundaries learned from less noisy training data, and are wrongly classified.


Note that perturbations in Figs. <ref> and <ref> are generated with total L_2-norm ϵ=3.0 for each trained network and at each SNR value. It can be seen from Fig. <ref> that adding random perturbations does not reduce the modulation-classification accuracy, yielding similar performance to the case when no defense mechanism is applied (Fig. <ref>).


When is employed, if the network is trained for a low SNR value, then test data with lower noise level (higher SNR) will lie at a larger distance from the decision boundaries (learned from noisy data), since the decision boundaries are already accounting for a very high noise level. Therefore, in this case the total perturbation ϵ may not be enough to move the signal to the wrong side of a learned decision boundary of the intruder, resulting in a higher accuracy. On the other hand, when the network is trained for a higher SNR value than the test channel, there is not much variation in the training data due to the absence of noise, and an attacks with even limited perturbation is enough to move the data point to the other side of the learned decision boundary, changing the class label.

In case of , the intruder networks trained at low channel SNR values of 0 dB and 10 dB are more robust against  as the learned decision boundary accounts for larger channel noise during training of the intruder NN and perturbation norm ϵ is too small in comparison to channel noise at smaller test SNR values to move the perturbed signal across it. Once the defensive perturbation ϵ becomes comparable in magnitude to the test data channel SNR then both intruder networks show similar performance for test SNR (≥ 5 DB). In the case of an intruder network trained at SNR 20 dB, perturbation ϵ is large compared to channel noise for higher test SNR values, and thus, results in small modulation-detection accuracy. Also, since the signal is perturbed before transmission, these defensive perturbations are partially masked by the channel noise. This effect of the channel noise is prominent in accuracy curves, though perturbations significantly reduce the detection accuracy as evident in Fig. <ref>.






 §.§ Improving intruder's performance by diversifying the training data


In this section, we consider the scenario when the intruder has training data available at different SNR values ranging from -20 dB to 20 dB. We consider a larger training set consisting of training samples for diverse channel SNR values (21 different SNR levels uniformly spaced between -20 dB to 20 dB), leading to a total of 21 × 12966 samples. We consider two different training strategies: (i) randomly shuffle the training data of all channel SNR values to train the intruder's DNN; and (ii) curriculum learning <cit.>, where the training data is arranged in descending order of their SNR values, and the training is started with samples of training data from SNR 20 dB, gradually adding samples with lower SNR values.



Fig. <ref> shows that an intruder network trained with data from all SNR values achieves a higher modulation-classification accuracy for and against all defensive modulation strategies compared to the case when only samples from the same SNR values were used (cf. Fig. <ref>); this is most likely due to the approximately 20-fold increase in the number of training samples used. On the other hand, we can see in Fig. <ref> that curriculum training achieves even higher robustness against all the defensive modulation schemes, and even the idealized defensive modulation scheme can be detected with more than 60% accuracy. This is because, in curriculum training, the neural network gradually learns, starting from easier concepts to more complex ones (more noisy channels in our case) and generalizes better to unseen data including those generated by defensive modulation schemes. In both cases, the improvement in detection accuracy is more for higher SNR values. Fig. <ref> shows the BER for 64 when defensive perturbations are used against these intruder modulation classifiers trained over the whole range of SNR values without and with curriculum training, respectively. The achieved BERs are similar to those achieved when the intruder classifiers are trained for a particular SNR in Fig. <ref>. This shows that the comparison of the detection accuracy discussed above is fair (i.e., the improved detection accuracy is not because the applied defensive perturbations are smaller).


Next, we consider the performance of BER-aware defensive modulation schemes when the intruder classifiers are trained with complete training data of all channel SNR values. The results without any curriculum learning are shown in Fig. <ref> for the same DNN-based classifier.  It can be seen that the modulation-classification accuracy is quite high, around 95%, when no defense mechanism is employed (), and over 90% when only noise is added (). We can also observe that, compared to results in Fig. <ref>, is less successful against this model for large λ (10^6); on the other hand, the BER is significantly improved, as demonstrated by comparing Fig. <ref> and Fig. <ref>. There is also a significant improvement in detection accuracy for essentially the same BER compared to the case when only training data for the same SNR value is used (cf. Fig. <ref> and Fig. <ref>).

On the other hand, using this larger set of training data yields no significant improvement in the performance of the tree-based classifier, and the results are very similar to those reported in Fig. <ref> (hence, they are omitted).







When the DNN-based classifier is trained using the complete dataset with curriculum learning, a significantly higher modulation-classification accuracy can be achieved against all defensive modulation schemes, as shown in  Fig. <ref>. Compared to the non-curriculum learning results in Fig. <ref>, we can see that the improved detection accuracy also results in a smaller BER.
This suggests that, for a fair comparison between the two approaches, we can increase the attack strength in the case of curriculum learning until we achieve similar BER values as in Fig. <ref>.



To this end, we increase the norm of perturbations for the scheme against the DNN-based intruder network trained with curriculum learning. Note that to make the defense mechanisms work, we need to increase the value of λ, and we have found that (the surprisingly large) λ = 10^20 works well in our experiments. The results are shown in Fig. <ref>. It can be seen that defensive perturbations with larger norms decrease the modulation-detection accuracy of the intruder, but they also result in significantly higher BER despite the very large λ value.

The results in this section showed that using more and diverse data and curriculum training can significantly improve the performance of the intruder and its robustness against various defense mechanisms. While designing better defense mechanisms against these intruders is an interesting and challenging future research direction, one method that can be employed directly at the transmitter is to reduce the code rate, which allows employing stronger attacks at the transmitter. This is explored in the next section.





 §.§ The effect of the code rate


In our previous experiments we considered a fixed code rate of 2/3. However, we have observed that the BER increases significantly for some of the modulation schemes due to input perturbations. One way to reduce the BER in the presence of the defensive perturbations to the transmitted signal is to introduce additional redundancy by decreasing the code rate.

To illustrate the effect of the code rate, we first evaluate the performance of our BER-aware defense schemes (with ϵ = 3) for a channel code of rate 1/2 against the usual DNN-based intruder trained for a specific SNR. The results, shown in Fig. <ref> demonstrate that both the BER and the detection accuracy can be substantially reduced compare to the case when the code rate is 2/3 (see Fig. <ref> and Fig. <ref> for comparison). For example, even for 64, (with λ= 10^6) achieves zero BER for high SNR values (at least 16 dB).





The very small BERs (obtained in the previous experiment) allow the application of more aggressive
defensive perturbations when the intruder employs a stronger classifier. Accordingly, we evaluate the defensive scheme (with a large λ =10^20) for different perturbation norms against a DNN-based intruder trained with curriculum learning over a range of SNR values (the setup is the same as for Fig. <ref> except for the code rate). The results, shown in Fig. <ref>, demonstrate that, compared to Fig. <ref>, using a lower code rate of 1/2, the modulation-classification accuracy of the intruder trained with curriculum learning can be reduced without incurring a large BER at the legitimate receiver.






§ CONCLUSIONS AND FUTURE WORK



We proposed a novel approach to secure wireless communication by preventing an intruder
from detecting the modulation scheme employed, which is typically the first step of a more advanced attack. In the proposed scheme, the I/Q symbols of the modulated waveform at the transmitter are perturbed using an adversarial perturbation derived against the modulation classifier of the intruder. The perturbation is designed using PGD, whose goal is to identify a perturbation with a limited norm that is sufficient to fool the intruder's classifier. More advanced methods are also proposed, whose goal is also to keep small the BER caused by the perturbation at the legitimate receiver. Experimental results verify the viability of our approach by showing that our methods are able to substantially reduce the modulation-classification accuracy of the intruder with minimal sacrifice in the communication performance. We have also shown that the intruder can improve its detection accuracy significantly by training with a dataset of samples taken from a range of SNR values, especially when curriculum learning is also employed. This provides robustness against channel noise as well as potential defense mechanisms against the intruder, and has led to improvements upon state-of-the-art modulation detectors in our experiments. Finally we have shown that a better trade-off between the intruder's detection accuracy and the BER at the legitimate receiver can be achieved by sacrificing the communication rate.

Utilizing the fast advances in the field of adversarial machine learning, our defense methods can certainly be improved in the future by applying more advanced as well as more universal (e.g., black box) adversarial attack methods. Another interesting avenue for future research is to develop sophisticated defensive perturbations that can exploit different channel characteristics both at the intruder and legitimate receiver. On the other end of the problem, one can develop better training strategies for the intruder that can achieve more robust performance against these defense mechanisms, for example, by applying adversarial training methods <cit.>.



IEEEtran



