A Comprehensive Analysis of Information Leakage in Deep Transfer Learning

Transfer learning is widely used for transferring knowledge from a source domain to the target domain where the labeled data is scarce. Recently, deep transfer learning has achieved remarkable progress in various applications. However, the source and target datasets usually belong to two different organizations in many real-world scenarios, potential privacy issues in deep transfer learning are posed. In this study, to thoroughly analyze the potential privacy leakage in deep transfer learning, we first divide previous methods into three categories. Based on that, we demonstrate specific threats that lead to unintentional privacy leakage in each category. Additionally, we also provide some solutions to prevent these threats. To the best of our knowledge, our study is the first to provide a thorough analysis of the information leakage issues in deep transfer learning methods and provide potential solutions to the issue. Extensive experiments on two public datasets and an industry dataset are conducted to show the privacy leakage under different deep transfer learning settings and defense solution effectiveness.

A fundamental block of deep transfer learning is deep neural network, which is vulnerable to different attacks aiming to detect sensitive information contained in the training dataset [7,32,37]. Moreover, in most of the real-world applications where deep transfer learning is used, the source and target datasets always reside in two different organizations. As a result, deep transfer learning also faces potential privacy threats, i.e, the client in the target organization can leverage the vulnerability of deep learning models to detect sensitive information contained in the source organization. Specifically, applying deep transfer learning comes with the interaction between the source and target domains. Thus, the data transmission between these domains may unintentionally disclose private information.
Existing studies on analyzing privacy leakages focus on either general machine learning models [25,32] or in a federated learning setting where model is collaboratively trained by multiple clients by sharing and aggregating the gradients via a server [22,25]. However, there no such study on transfer learning paradigms. To this end, we are the first to provide a general categorization for deep transfer learning models based on the potential information leakages. This is not trivial since there are numerous methods for deep transfer learning [27,35,40]. Given the goal of privacy leakage analysis, we care more about the interaction manner between source and target domains. Thus, we divide previous works into three categories, as illustrated in Figure 1: (1) model-based paradigm where the whole model structure and parameters are shared (2) mappingbased where the hidden features are shared (3) parameter-based where the parameter gradients are shared. Based on that, the previous works can fall into the above categories or a hybrid of them. For example, fine-tuning based approaches obviously belong to the first category. The prior work [34] is based on the mapping-based paradigm, since it uses the correlation alignment loss which further depends on the shared hidden features. Similarly, previous works that minimize the domain representation difference by variants of distribution divergence metrics such as maximum mean discrepancy also fall into the second category [17,18,29]. Fully-shared and shared-private transfer learning models [15] can be regarded as parameter-based, as they both jointly train a shared network via gradient updates in a multi-task fashion, just to name a few.
Based on the general categorization, we can build customized attacks against each paradigm and demonstrate information leakages in deep transfer learning. At a high level, we consider inferring two types of sensitive information, i.e., membership and property information. This sensitive information can be revealed by the transmission data between the two domains as the above discussed. Co-training hM s i < l a t e x i t s h a 1 _ b a s e 6 4 = " q z 7 r B a U e M 7 t E 5 z 3 m 4 B h u 8 + l 5 I j w = " > A A A C 7 n i c j V H L S s N A F D 3 G 9 7 v q 0 k 2 w C L o p q R R 0 W X T j R l C w W j B S J n H a D k 4 e T C Z C K f 0 G d + 7 E r T / g V j 9 D / A P 9 C + + M K f h A d E K S M + f e c 2 b u v U E q R a Y 9 7 2 X E G R 0 b n 5 i c m p 6 Z n Z t f W C w t L Z 9 k S a 5 C 3 g g T m a h m w D I u R c w b W m j J m 6 n i L A o k P w 0 u 9 0 z 8 9 I q r T C T x s e 6 l / D x i n V i 0 R c g 0 U a 3 S p i 9 T h E j u k b 5 9 2 r Z Q N a K 8 9 Y 6 P 2 6 R R B r y S l j X 3 S h J Q n C e v T b B N P j L N m f / O e G E 9 9 t z H 9 v d R r R K z C g N i / d N P M / + p 0 L Q o 9 H J s a O N U U G U Z X 5 6 c u i e m K v r n 9 p S p F D h F x G n c p L g n 7 R j n t s 2 0 0 s a l d 9 9 Y 1 8 T e T q V m 9 9 9 P c B O / 6 l j T g 0 s 9 x z o L 6 Y b F U L p b P y 4 X K S T r q L H a w h w O a 5 x E q O E M V N f K + w S O e 8 G w x 6 9 a 6 s + 4 / U 6 1 M q t n G t 2 U 9 f A A N v p t s < / l a t e x i t > hwi < l a t e x i t s h a 1 _ b a s e 6 4 l l c c r w i T 2 n b 4 9 2 R z k b 0 d 5 4 p l Y d 0 i m S X k V K h g 3 S x J S n C J v T m I 1 n 1 t m w v 3 k P r a e 5 2 4 D + Q e 7 V J 1 b j j N i / d K P M / + p M L R p d 7 N g a B N W U W M Z U F + Y u m e 2 K u T n 7 V J U m h 4 Q 4 g 0 8 p r g i H V j n q M 7 O a 1 N Z u e u v b + K v N N K z Z h 3 l u h j d z S x p w 5 f s 4 f 4 L m V r l S L V c P q q X a b j 7 q I l a x j k 2 a 5 z Z q 2 E c d D f K + x g M e 8 e R 0 n R v n 1 r n 7 S H U K u W Y F X 5 Z z / w 6 r x Z u l < / l a t e x i t > hwi < l a t e x i t s h a 1 _ b a s e 6 4 l l c c r w i T 2 n b 4 9 2 R z k b 0 d 5 4 p l Y d 0 i m S X k V K h g 3 S x J S n C J v T m I 1 n 1 t m w v 3 k P r a e 5 2 4 D + Q e 7 V J 1 b j j N i / d K P M / + p M L R p d 7 N g a B N W U W M Z U F + Y u m e 2 K u T n 7 V J U m h 4 Q 4 g 0 8 p r g i H V j n q M 7 O a 1 N Z u e u v b + K v N N K z Z h 3 l u h j d z S x p w 5 f s 4 f 4 L m V r l S L V c P q q X a b j 7 q I l a x j k 2 a 5 z Z q 2 E c d D f K + x g M e 8 e R 0 n R v n 1 r n 7 S H U K u W Y F X 5 Z z / w 6 r x Z u l < / l a t e x i t > hhi < l a t e x i t s h a 1 _ b a s e 6 4 = " f B B e 8 T h E j u k b 5 9 2 r Z Q N a K 8 9 Y 6 P 2 6 R R B r y S l j X 3 S h J Q n C e v T b B N P j L N m f / O e G E 9 9 t z H 9 v d R r R K z C g N i / d N P M / + p 0 L Q o 9 H J s a O N U U G U Z X 5 6 c u i e m K v r n 9 p S p F D h F x G n c p L g n 7 R j n t s 2 0 0 s a l d 9 9 Y 1 8 T e T q V m 9 9 9 P c B O / 6 l j T g 0 s 9 x z o L 6 Y b F U L p b P y 4 X K S T r q L H a w h w O a 5 x E q O E M V N f K + w S O e 8 G w x 6 9 a 6 s + 4 / U 6 1 M q t n G t 2 U 9 f A A N v p t s < / l a t e x i t > x U e 8 I g n 4 8 y 4 M W 6 N u 4 9 U I 5 N q N v F l G f f v a 2 2 U q A = = < / l a t e x i t > D S < l a t e x i t s h a 1 _ b a s e 6 4 = " c f E t a r D u W x T b t X 4 K l G t v P x n o J / I = " > A A A C 0 H i c j V H L S g M x F D 0 d X 7 W + q i 7 d D B b B V Z l K Q Z d F X b i s j 7 a C r Z K J U Y P z c i Y j S h F x 6 w + 4 1 a 8 S / 0 D / w p s Y Q S 2 i G W b m 5 N x 7 T n L v 9 Z N A Z s r z X g r O 0 P D I 6 F h x v D Q x O T U 9 U 5 6 d a 2 d x n n L R 4 n E Q p / s + y 0 Q g I 9 F S U g V i P 0 k F C / 1 A d P z z D R 3 v X I o 0 k 3 G 0 p 6 4 T 0 Q v Z a S R P J G e K q F 4 3 Z O q M s 6 C / e X O 4 e 1 S u e F X P L H c Q 1 C y o w K 5 m X H 5 G F 8 e I w Z E j h E A E R T g A Q 0 b P A W r w k B D X Q 5 + 4 l J A 0 c Y E b l E i b U 5 a g D E b s O X 1 P a X d g 2 Y j 2 2 j M z a k 6 n B P S m p H S x R J q Y 8 l L C + j T X x H P j r N n f v P v G U 9 / t m v 6 + 9 Q q J V T g j 9 i / d Z x U e 8 I g n 4 8 y 4 M W 6 N u 4 9 U I 5 N q N v F l G f f v a 2 2 U q A = = < / l a t e x i t > D S < l a t e x i t s h a 1 _ b a s e 6 4 = " c f E t a r D u W x T b t X 4 K l G t v P x n o J / I = " > A A A C 0 H i c j V H L S g M x F D 0 d X 7 W + q i 7 d D B b B V Z l K Q Z d F X b i s j 7 a C r Z K J U Y P z c i Y j S h F x 6 w + 4 1 a 8 S / 0 D / w p s Y Q S 2 i G W b m 5 N x 7 T n L v 9 Z N A Z s r z X g r O 0 P D I 6 F h x v D Q x O T U 9 U 5 6 d a 2 d x n n L R 4 n E Q p / s + y 0 Q g I 9 F S U g V i P 0 k F C / 1 A d P z z D R 3 v X I o 0 k 3 G 0 p 6 4 T 0 Q v Z a S R P J G e K q F 4 3 Z O q M s 6 C / e X O 4 e 1 S u e F X P L H c Q 1 C y o w K 5 m X H 5 G F 8 e I w Z E j h E A E R T g A Q 0 b P A W r w k B D X Q 5 + 4 l J A 0 c Y E b l E i b U 5 a g D E b s O X 1 P a X d g 2 Y j 2 2 j M z a k 6 n B P S m p H S x R J q Y 8 l L C + j T X x H P j r N n f v P v G U 9 / t m v 6 + 9 Q q J V T g j 9 i / d Z  attack which takes the model (learned using the source dataset) as input and determines whether a specific sample is used for training the model. In the mapping-based setting, we can build the property attack to infer properties contained in the training dataset. For example, the attacker resides on the target domain aims to infer properties of the source domain based on the shared hidden features. In the parameter-based setting, we can similarly perform the property inference attack, i.e., the attacker can infer properties of the source domain data based on the shared gradients. More details of these attacks can be found in Section 3. Empirically, to demonstrate the effectiveness of attacks, we conduct a set of experiments under the three types of transfer learning settings. Our key observation is that all these types of models do unintentionally leak information of the training data under membership/property attacks. Model-based paradigm is possible to leak membership information. Parameter-based paradigm without revealing individual gradient (i.e., averaged gradients at batch-level) leaks much less property information, compared to the mappingbased paradigm where hidden features (i.e., representations at sample-level) are shared.
In summary, our main contributions are as follows: • We are the first to propose a general categorization for different deep transfer learning paradigms based on their intrinsic interaction manner between the source and target domains and provide a comprehensive analysis of the potential privacy leakage profiles for each category respectively. • Based on the categorization, we build specific attacks against each paradigm to demonstrate their privacy leakages. • In contrast to previous works that aim to design privacypreserving transfer learning model for a specific setting (e.g., mapping-based [16,31]), our analysis has covered a wide range of deep transfer learning methods. • We conduct extensive experiments on both public datasets and an industry marketing dataset to verify the effectiveness of our attacks and defense solutions.

PRELIMINARIES 2.1 Deep Transfer Learning Setting
In this work, we focus on deep transfer learning [35,40], where the models discussed are neural network based. Without losing generality, we consider a transfer learning setting with two domain tasks T ∈ {S,T }, where S and T refer to the source domain and target domain respectively, both containing private sensitive information. We aim to improve the target domain learning task performance by utilizing its own data D T and source domain data D S . Each dataset contains a set of labeled examples z i = (x i , y i ), where x denotes the inputs and y denotes the corresponding labels. The size of source domain data D S is usually much larger than the target domain data D T , i.e., N S >> N T . The goal of a domain task is to learning a transformation function f w (x, y) : R d → R, parameterized by model weights w.

Inference Attacks for DNN Models
The basic idea of the inference attack is to exploit the leakages when a model is being trained or released to reveal some unintended information from the training data. In this section, we briefly present two types of inference attacks, i.e., membership and property attacks, for machine learning models. Membership Inference Attack. Membership inference is a typical attack that aims to determine whether a sample is used as part of the training dataset. Membership inference may reveal sensitive information that leads to privacy breach. For example, if we can infer a patient's existence in the training dataset for a medical study of a certain disease, we can probably claim that this patient has such a disease. Recent works have demonstrated the membership attack attempts for machine learning models under the black-box setting [19,30,32], white-box setting [22,26] or both [9]. Shadow training is a widely adopted technique for membership inference, where multiple shadow models are trained to mimic the behavior of the target model [19,30,32]. This technique assumes the attacker to have some prior knowledge about the population from the targeted model training dataset was drawn. Recent works by [22,26] explicitly exploit the vulnerabilities in gradient updates to perform attacks with white-box access. Property Inference Attack. Another common type of attack is property inference that aims to reveal certain unintended or sensitive properties (e.g., the fraction of the data belongs to a certain minority group) of the participating training datasets that the model producer does not intend to share when the model is released. A property is usually uncorrelated or loosely correlated with the main training task. Pioneer works of [2,6,7] conducted the property attacks that characterize the entire training dataset. While, [22] aimed to infer properties for a subset of the training inputs, i.e., in terms of single batches which they termed as single-batch properties. In this regard, membership attack can be viewed as a special case of property attack when scope for property attack is on a sample basis. The above-mentioned two types of attacks are closely related. Most of existing works perform those attacks against general machine learning models, while a few focus on the federated learning and collaborative learning scenarios [22,26]. None of the studies systematically explore the inference attacks explicitly for the context of deep transfer learning.

PRIVACY LEAKAGE IN DEEP TRANSFER LEARNING
In this section, we first provide a general categorization of deep transfer learning according to the interaction manner between source and target domains. Then, based on the categorization, we conduct privacy analysis through building specific attacks against different transfer learning paradigms.

General Categorization of Deep Transfer Learning
Different types of transfer learning models have been proposed over the years, depending on how the knowledge is shared. Although in transfer learning, there may exist several existing categorizations in the literature [25,27,35], we categorize the different deep transfer learning models based on how the two domains interact, their training strategy (e.g., co-training or self-training), and the potential leakages.
Broadly speaking, those transfer learning models can be categorized into three types, i.e., model-based, mapping-based, and parameterbased, as illustrated in Figure 1.
• Model-based (in Figure 1(a)) is a simple but effective transfer learning paradigm, where the pre-trained source domain model is used as the initialization for continued training on the target domain data. Model-based fine-tuning has been broadly used to reduce the number of labeled data needed for learning new tasks/tasks of new domains [4,10,11]. • Mapping-based (in Figure 1(b)) methods aim to align the hidden representations by explicitly reducing the marginal or conditional distribution divergence between source and target domains. More specifically, alignment losses, i.e., usually in forms of distribution discrepancies/distance metrics such as MMD and JMMD, between domains are measured by such feature mapping approaches and minimized in the loss functions [17,18,29].
• Parameter-based (in Figure 1(c)) methods transfer knowledge by jointly updating a shared partial network to learn transferable feature representations across domains in a multi-task fashion. This type of methods is mainly achieved by parameter sharing. Regardless of the design differences, they all utilize a shared network structure to transform the input data into domain-invariant hidden representations [15,38,39]. Based on the general categorization, we will discuss the threat model, information leakage and the customized attacks against each transfer learning paradigm in detail.

Threat Model
In this work, we assume all the parties , i.e., the domain-specific data owners, are semi-honest, where they follow exactly the computation protocols but may try to infer as much information as possible when interacting with the other parties. More specifically, we work on the threat model under a deep transfer learning setting with two domains. Without loss of generality, we assume the owner of the target dataset to be the attacker, who intentionally attempts to infer additional information from source domain data beyond what is explicitly revealed. Depending on the different deep transfer learning categorizations, the attacker may have different access to the source domain information. Note that we can naturally extend to the case where the attacker is the owner of the source dataset or the transfer learning setting with more than two data sources, however, it is not the focus of this paper.

Privacy Analysis
As presented in Table 1, three types of model categorization require different information interactions and training strategies, thus posing different potential leakage profiles. More specifically: • Model-based methods rely on the pre-trained source domain model solely. Despite the necessity for a pre-trained model to be disclosed to the target domain, both the source and target training processes can be entirely separately, thus the potential leakage is only the final source domain model M S . In this case, the attacker has full access to the source domain model including both the model structure and parameters. • Mapping-based methods are optimized in a co-training fashion and hidden presentations of both domains have to interact with each other to measure the alignment losses or domain regularizers. We denote h S ik and h T ik as the hidden representation of layer i at a training iteration k ∈ [1, 2, ..., K] for source and target domains, respectively. Specifically, such a feature matching process demands the hidden presentations, i.e., h S ik and h T ik , of both domains to be exposed and aligned to reduce the marginal or conditional distribution divergence between domains. As a result, h S , h T can potentially leak information from the training data.  Co-training ⟨w c ⟩ Batch property (either source or target). In such an alternating process, w c k has to be revealed across domains at each iteration k ∈ [1, 2, ..., K]. Thus, the potential leakage profile w c k contains all the intermediate w c k during the training process. As a summary, in this paper, we focus on inferring information that is unintentionally disclosed by the data transmission between the source and target domains, such as membership information of an individual sample and property information of a specific sample or a subset of samples (see more details in the next subsection.).

Inference Attacks for Deep Transfer Learning
Inference attacks against deep learning models generally exploit implicit memorization in the models to recover sensitive information from the training data. Previously studies have proven that information can be inferred from the leakage profile, such as membership information that decides whether a sample is used for training [14,20,22,32,36], properties which reveals certain data characteristics, or reconstructed feature values of the private training records. In this part, to empirically evaluate the privacy leakage in different transfer learning settings (shown in Figure 1), we present concrete attack methods for each setting. Note, we assume the attacker to be the owner of the target dataset for all the three transfer learning paradigms discussed above. Model-based. As illustrated in Table 1, the only leakage source in this setting is the model trained using data from the source domain. For simplicity, we denote the trained model as M S : y = f S w (x), where y is the prediction label. According to the training protocol in the model-based setting, the attacker can obtain the white-box access of M S , i.e., its structure and parameters. Thus, based on recent works [19,22,26,30,32], an attacker can design powerful membership attack methods to detect sensitive information contained in the source domain (i.e., membership attack). In the context of membership attack, the goal of the attacker is to train a membership predictor, which can take a given sample x as input and output the probability distribution that indicates whether the given sample is used for training the source domain model M S . Formally, we denote the membership predictor as A mem : x, M S → {P(m = 1|x), P(m = 0|x)}. Here, m = 1 means the given input x is used for training the model M S .
In this paper, we employ a widely used technique named shadow model training for building the membership predictor. Specifically, we assume the attacker can have extra knowledge about the source dataset D S . The extra prior knowledge in our setting is the shadow training dataset D shadow that comes from the same underlying distribution as D S for training the source model. The core idea of the shadow training dataset is to first train multiple shadow models that mimic the behavior of the source model. Then, the attacker can extract useful features from the shadow models/datasets and build a machine learning model that characterizes the relationship between the extracted features and the membership information.
To be specific, the attacker first evenly divides the shadow training dataset D shadow into two disjoint datasets D t r ain shadow and D out shadow . Then the attacker trains the shadow model that has the same architecture as the source model using D shadow . Subsequently, features of samples from both D t r ain shadow and D out shadow can be extracted. For each sample in D shadow , the attacker can use the output prediction vector of the shadow model as the feature following the prior work [32]. And each feature vector is labeled with 1 (member, if the sample is in D t r ain shadow ) or 0 (non-member, if the sample is in D out shadow ). At last, all the feature-label pairs are used for training the membership predictor.
Once the membership predictor is obtained, we can use it to predict the membership label of a sample in D S , i.e., feeding the output vector of the source domain model to the predictor. Mapping-based. In the mapping-based setting, information leakage can be both posed at the source and target domains since the training protocol proceeds in an interactive manner. To keep the analysis consistency, we refer the attacker to be the owner of the target domain dataset.
The core idea of the mapping-based method is to align the hidden features extracted from the source and target domains, i.e., reducing the discrepancy between feature distributions of these two domains. This can be done by minimizing some pre-defined alignment loss function (e.g., maximum mean discrepancy [17,18,25,29]). Thus, this method will lead to the hidden features of the source domain share a similar distribution to the features of the target domain, which comes with a strong privacy implication, i.e., the attacker can leverage the feature similarity to build the attack model to detect sensitive information contained in the source domain.
We consider the property attack [7] in this setting. At a high level, the property attack aims to infer whether a feature comes from the source domain has a specific property or not. Here, we take the case of the k-th iteration as an example. Given a specific property, the attacker can first collect an auxiliary dataset for assisting the attack. Specifically, the attacker divides the target domain dataset D T into two subsets, namely, D T pr op which contains samples with the property and D T nonpr op which consists of samples without the property. Subsequently, D T pr op and D T nonpr op are used as the auxiliary datasets for building the attack model. At the k-th iteration, given the current parameter w k of the model trained using the target dataset, the attacker calculates the hidden features h T k of the alignment layer k with respect to the samples in the auxiliary dataset. Hidden features are labeled with 1 if the corresponding sample is in D T pr op and 0 if the sample is in D T nonpr op . Once the attacker collects all the feature-label pairs, she can train a property predictor A pr op using these pairs. The whole procedure is demonstrated in Algorithm 1.
Based on the property predictor obtained above, the attacker can conduct an online attack. Specifically, in the joint training process, at the beginning of the k-th iteration, the attacker receives a batch of hidden features h S k from the source domain. Then, the attacker can employ A pr op to predict the property information contained in the source domain dataset. Parameter-based. In the parameter-based setting, information leakage is posed by the weight parameter interaction between the source and target domains. Similar to previous settings, we refer the attacker to be the owner of the dataset of the target domain.
We consider the batch property attack in this setting. The intuition behind this attack is that the attacker can observe the updates of the share layers calculated using a mini-batch of samples from the source domain. Thus the attacker can train a batch property predictor A bpr op to infer whether the update based on the minibatch with a given property or not. We take the k-th iteration as an example to demonstrate how the attacker conducts the attack. We assume that the attacker has auxiliary dataset D aux consisting of samples from the distribution that is similar to the source domain distribution. Note that, this assumption is commonly used in various previous works [23,31]. Given a specific property, the attacker can further divide D aux into two sub-datasets, namely, the dataset which has the property (D aux pr op ) and the dataset without the property (D aux nonpr op ). Based on the above setting, at the k-th iteration, the attacker can receive the fresh parameter w k of the shared layer which is updated based on samples from the source dataset. Then, the attacker can calculate gradients д pr op using the mini-batch sampled from D aux pr op and д nonpr op based on samples from D aux nonpr op . The batch gradients based on D aux pr op are labeled with 1 and others are labeled with 0. Based on these gradient-label pairs, we can train the batch property predictor A bpr op . The whole procedure is shown in Algorithm 2.
Once A bpr op is obtained, the attacker can use it to predict the btach property information of the source domain. Specifically, the attacker first generates the gradient д online based on the parameters of the current and last iteration (i.e., w k and w k−1 ). Then she can feed д online to A bpr op to obtain the prediction result.

EXPERIMENTS
In this section, we empirically study the potential information leakages in deep transfer learning. We start by describing the datasets and the experimental setup. We then discuss in detail the results for various inference attacks under the aforementioned three transfer learning settings, followed by the examination of viable defenses.

Datasets and Experimental Setup
We conduct experiments on two public and one industry datasets, i.e., Review, UCI-Adult, and Marketing. The Review dataset is used to examine membership attack as fine-tuning methods are often used in such NLP task and typically suffer from membership inference. While the UCI-Adult dataset has some sensitive properties, thus is used for conducting property based attacks. Besides, the Marketing dataset is sampled from real-world marketing campaigns to examine the defense solution. Data statistics are in Table 2. Review. We construct our dataset from the public available Amazon review data [21], with two categories of products selected, i.e., 'Watches' and 'Electronics. In our transfer learning setting, the data-abundant domain 'Electronics' is viewed as the source domain, while the data-insufficient domain 'Watches' is treated as the target. Each sample consists of a quality score and a review text. Following the literature on this task [3], we adopt the TextCNN [12] as the base model for textual representation in the transfer learning model. For TextCNN, we set the filter windows as 2,3,4,5 with 128 feature maps each. The max sequence length is set as 60 for the task. We initialize the embedding lookup table with pre-train word embedding from GloVe with embedding dimension set as 300. The batch sizes for training the source domain model, the shadow model, and the attack model are set as 64. UCI-Adult. We consider a second Adult Census Income dataset [13] from the UC Irvine repository to examine the property attack issues. The dataset has 14 properties such as country, age, work-class, education, etc. To form the transfer learning datasets, we use data instances with country of "U.S." as source domain and "non-U.S." as the target. For the UCI dataset, we consider an MLP as our base model for the transfer learning models. Marketing. For transfer learning purpose, two sets of data are sampled from two real-world marketing campaigns data that contains user profile, behaviour features and adoptions. One data-abundant campaign is used as source domain, while the other is used as the target. The task is to predict whether a user will adopt the coupon. Implementation Details. Specifically, for model-based setting, a fine-tuning approach is adopted where both the source domain model and the shadow model employed the same model structure, i.e., the above-mentioned TextCNN structure, followed by an MLP layer of size 128. The attack model adopted is an MLP with network of [16,8]. For the mapping-based setting, we consider an MLP with a network structure of [64,8] for source and target models, an MMD metric is used as the alignment loss between the two domains, and another MLP with network of [64,8] for the attack model. For parameter-based setting, we consider a fully-shared model structure with an MLP as the base model with structure of [64,8] at the task training stage, and another MLP with network of [16,8] for the attack model.
All the above neural network based transfer learning models are implemented with TensorFlow and trained with NVIDIA Tesla P100 GPU using Adam optimizer, if not specified. The learning rate is set as 0.001. The activation function is set as ReLU. For all the settings, we use Area Under Curve (AUC) to evaluate the overall attack performance. For membership attack, we also adopt precision for evaluation, as membership of the training dataset is the concern. Precision is defined as the fraction of samples predicted as members are indeed members of the training data. For property attacks, we further include accuracy for evaluating the overall attack performance.

Model-based Setting
In the model-based setting, the attacker has the full white-box access to the source model, thus the attacker is able to train shadow models to mimic the behavior of the source domain mode. In this setting, we explore the possibility of performing membership inference on the source model. Despite the context differences, membership attacks for model-based transfer learning models can be conducted the same way as for any trained stand-alone deep learning models [23,30,32].
To build the membership predictor introduced in Section 3, we split the original source dataset into three parts, namely, training dataset (248,011 samples), test dataset (70,860 samples), and shadow dataset (70,860 samples). We set the number of shadow models to 3. Table 3 shows the training and testing accuracy for the attack classifier on the shadow dataset and the attack performance, i.e., AUC and precision, on the source domain model. Smaller gaps between training and testing accuracy suggests better generalization and predictive power of the attack classifier. Overall, we find that even in the model-based setting without direct interactions between the source and target domains during the training process, the source domain model does leak a considerable amount of membership information. We also examine the effect of over-fitting by adjusting the number of epoch trained for both the source domain and shadow models. We observe the more over-fitted a model, the more information can be potentially leaked. However, it decreases when the number of epochs furthers increases. This echos the findings in [32] that over-fitting is an important factor but not the only factor that contributes to the leakage of membership information.

Mapping-based Setting
We then investigate property attacks under the mapping-based transfer learning setting to infer the properties of the source domain data during the training process. The properties are not the same as the prediction label, or even independent of the prediction task. To examine this, we construct two more datasets based on the UCI adult dataset: Prop-race and Prop-sex, discussed as follows.
• Prop-sex: we filter out the property "sex" feature from the input features in UCI-Adult data and use it as the attack task label. The attack label is set as 1 if the property "sex" is male   and 0 otherwise. This is to examine whether a attacker can infer training instance's gender during the training process. • Prop-race: similarly we filter out the "race" feature to study whether this property appears in training instances. Attack label is set as 1 if property "race" is white and 0 otherwise. Table 4 shows the results of the property attack on the Propsex dataset. The attack has a good AUC of 0.77 in the property inference task, which shows the transfer learning setting does have information leakage. The attack precision is high on the property "sex:male", and less satisfactory on "sex:female". The female property has much less instances and thus may be harder to be predicted.
We have also conducted experiments on another property "race" in Table 5, the attack AUC is lower than on Prop-sex, 0.5885 vs. 0.7766. These results demonstrate that Prop-sex leaks more property information than Prop-race in terms of attack AUC. At the first glance, "race: white" has much higher attack precision than "race:non-white", 0.8901 vs. 0.2179. Close examination shows "race: white" dominates the training data with around 80% label coverage.
We further examine the correlation between the chosen properties and the underlying Census Income prediction task, and compare it with the property attack results in Table 6. In general, both Prop-race and Prop-sex have information leakage issues, and the property with a higher negative/positive correlation with the main prediction task tends to cause higher potential information leakage.

Parameter-based Setting
For the parameter-based setting, we further process the UCI adult data to perform the batch property attack. Again we use these two properties "race" and "sex" and form two datasets namely BProprace and BProp-sex respectively. For both datasets, we set the batch size as 8. For each batch in BProp-race, we set the label as 1 if a batch has at least one data instance containing the "non-white" property  and 0 otherwise. For BProp-sex, we set the label as 1 if a batch has at least one instance containing the "female" property and 0 otherwise. The data statistics are shown in Table 7. We find for BProp-race, the positive instance ratio in the source domain is drastically different from the target, this may bring negative transfer. While for the BProp-sex, the positive instance ratio is similar for both domains. As shown in Algorithm 2, the selection of D aux is the key to building the batch property predictor. In this paper, we directly use the target dataset as the auxiliary dataset and find it works in practice, as both domains commonly are related for the transferring purpose. Generally, we can obtain better attack performance if we can use part of source domain samples to form the auxiliary dataset. The results are presented in Table 8. First, we find the attack AUC for the batch property attack in the parameter-based setting are generally not as high as those from the property attack in the mapping-based setting. The AUC is around 0.56 in batch property attack, while up to 0.77 for property attack. This shows the parameter-based setting is generally less vulnerable from attacks, which can possibly be justified by the fact that the gradients exchanged have been averaged at the batch-level. Second, the results for BProp-sex are better than BProp-race, as AUC results are 0.5654 and 0.5545 respectively. This is intuitive as observed from Table 7, the domain difference in BProp-race is larger than the BProp-sex. The model performance in BProp-race may suffer from the negative transfer learning due to the domain gap, thus may lead to the decrease in attack performance. Overall, the parameter-based method is possible to suffer from batch property attacks, however, the information leakage problem is not that severe.

Defense Solutions
A standard way to prevent statistical information leakage is differential privacy. However, the privacy guarantee provided by differential privacy comes with the decreasing of the model utility. Some recent works proposed to relax the requirement of differential privacy to prevent membership/property attacks while provide better model utility. For example, some regularization techniques such as dropout are helpful to prevent the information leakage of machine learning models [23,36]. Recent studies consider to use Stochastic gradient Langevin dynamics (SGLD) [24,36]. In this paper, to prevent the information leakage in deep transfer learning, we employ SGLD as our optimizer to optimize deep models and show its effectiveness in reducing information leakage. Prior work [36]  demonstrates that SGLD is effective in preventing membership information leakage and provides theoretical bounds for membership privacy loss. Empirically, in this paper, SGLD is also effective for preventing the leakage of the property information while provides comparable model performance compared with those non-private training methods.
To examine the effectiveness of the defense solutions, we conduct experiments under the mapping-based setting. Specifically, we replace the non-private optimizer (i.e., SGD) with SGLD and train the source and target models from scratch. The overall result is shown in Table 9. We observe that the use of SGLD can prevent the information leakage of the training dataset, based on the above attack metrics for evaluating the information leakage. In contrast to the original optimizer, SGLD significantly reduces the attack AUC score of the inference attack from 0.7766 to 0.6862 in Prop-sex, and 0.5885 to 0.5442 in Prop-race, while achieves better model utility in terms of the task AUC score especially in Prop-race (0.7750 to 0.8012). In both datasets, SGLD achieves the best task AUC and slightly outperforms the original task results. We infer the boost of the task AUC score is due to the regularization effect of the noises injected in SGLD, which can prevent model overfitting as pointed out in the previous works [24,36].
We also conduct experiments of DP-SGD introduced in the prior work [1]. As the result shows, although DP-SGD can achieve better anti-attack ability than SGLD, it comes with considerably model utility decrease (the task AUC score decreases from 0.8466 to 0.7662 in Prop-sex, and 0.7750 to 0.7192 in Prop-race ). Furthermore, experiments on Dropout are also performed. We can observe that Dropout can also prevent privacy leakage. We also note that when the drop ratio increases, the anti-attack ability of Dropout increases but model utility decreases drastically. As a conclusion, through experiments, we found that SGLD can be a good alternative to differentially private optimization methods and it can achieve a better trade-off between model utility and anti-attack ability.

INDUSTRIAL APPLICATION
Witness the effectiveness of the SGLD, we conduct additional experiments to examine whether it is able to prevent information leakage in an industrial application. The main task is to predict whether a user will use the coupon and the inference task is to predict the property âĂĲmarital status: marriedâĂİ (the Pearson correlation between these labels is -0.03986). Follow the original application, here we adopt a fully-shared model [15] under the parameter-based setting. As shown in Table 10, we find the original TL method does help to improve the target domain performance, boosting AUC from 0.7513 (Target-only) to 0.7704. However it also suffers from information leakage with an attack AUC of 0.5553. By combining SGLD with transfer learning, with a minor decrease of task performance (-0.8%), the information leakage can be significantly reduced by 3.8%.

RELATED WORK
Deep Transfer Learning Models. With the success of deep learning, deep transfer learning has been widely adopted in various applications [25,27,35]. According to our categorization based on the potential information leakage, transfer learning models can be summarized into three types, i.e., model-based, mapping-based, and parameter-based, or a hybrid of the different types [18].
Model-based methods, such as model fine-tuning, generally first pre-train a model using the source domain data and then continue training on the target domain data [4,10,11]. Mapping-based methods aim to align the hidden representations by explicitly reducing the marginal/conditional distribution divergence between source and target domains which are measured by some distribution difference metrics. Commonly used metrics include variants of Maximum Mean Discrepancy(MMD), Kullback-Leibler Divergence, Wasserstein distance, and etc [17,18,25,29,35]. Parameter-based methods transfer knowledge by jointly updating a shared network to learn domain-invariant features across domains, which is mainly achieved by parameter sharing [15,38,39]. Further works improve parameter-based methods by incorporating adversarial training to better learn domain-invariant shared representations [15].
There are few studies [16,31] for analyzing privacy leakages for general machine learning models or in a federated learning setting, however, there is no such privacy analysis work for transfer learning models. To bridge this gap, we first provide a general categorization of deep transfer learning models based on information leakage types and conduct thorough privacy analysis through building specific attacks against different transfer learning paradigms. We we also examine several general defense solutions to alleviate information leakage for the three paradigms. The privacy preserving models for transfer learning are rarely studied. Prior work [16] proposed a secure transfer learning algorithm under the mappingbased setting, where homomorphic encryption was adopted to ensure privacy at the expenses of efficiency and some model utility loss due to the Taylor approximation. A follow-up work in [31] further enhanced the security and efficient for the same problem setting by using Secret Sharing technique. A recent work by [8] employed a privacy preserving logistic regression with ϵ-differential privacy guarantee under a hypothesis transfer learning setting. Membership Inference. The study [32] developed the first membership attack against machine learning models with only black-box access using a shallow training method. This method assumes that we have some prior knowledge of the data for training the attack classifier is from the same distribution as the original training data. Later study of [19] followed the idea of shallow training and explored two more targeted membership attacks, i.e., frequency-based and distance-based. The study [30] further relaxed key attack assumptions of [32] and demonstrated more applicable attacks. Aside from the black-box setting, these studies [22,26] examined the membership attacks against federated/collaborative learning under the white-box setting, where an adversary can access the model and potentially is able to observe/eavesdrop the intermediate computations at hidden layers. They share a similar idea that leverages the gradients or model snapshots to produce the labeled examples for training a binary membership classifier. This work [9] presented the first membership attacks on both black-box and white-box for generative models, in particular generative adversarial networks (GANs). Property Inference. Property attack aims to infer the properties hold for the whole or certain subsets of the training data. Prior works [2,6,7] studied property inference attacks that characterize the entire training dataset. A property attack was developed [7] based on the concept of permutation invariance for fully connected neural networks, with the assumption that adversary has white-box knowledge. Concurrently, the study [22] developed attacks under the collaborative learning setting, where they focus on inferring properties for single batches of training inputs.

CONCLUSION
In this study, we provide a general categorization of different deep transfer learning paradigms depending on how the domains interact with each other. Based on that, we then analyze their respective privacy leakage profiles, design different attack models for each paradigm and provide potential solutions to prevent these threats. Extensive experiments have been conducted to examine the potential privacy leakage and effectiveness of defense solutions.