Question: Represent the article for finding a single-sentence TLDR

Question: In this paper, we explore new approaches to combining information encoded within the learned representations of autoencoders. We explore models that are capable of combining the attributes of multiple inputs such that a resynthesised output is trained to fool an adversarial discriminator for real versus synthesised data. Furthermore, we explore the use of such an architecture in the context of semi-supervised learning, where we learn a mixing function whose objective is to produce interpolations of hidden states, or masked combinations of latent representations that are consistent with a conditioned class label. We show quantitative and qualitative evidence that such a formulation is an interesting avenue of research. The autoencoder is a fundamental building block in unsupervised learning. Autoencoders are trained to reconstruct their inputs after being processed by two neural networks: an encoder which encodes the input to a high-level representation or bottleneck, and a decoder which performs the reconstruction using the representation as input. One primary goal of the autoencoder is to learn representations of the input data which are useful BID1, which may help in downstream tasks such as classification BID27 BID9 or reinforcement learning BID20 BID5. The representations of autoencoders can be encouraged to contain more'useful' information by restricting the size of the bottleneck, through the use of input noise (e.g., in denoising autoencoders, BID23, through regularisation of the encoder function BID17, or by introducing a prior BID11 . Another goal is in learning interpretable representations BID3 BID10 . In unsupervised learning, learning often involves qualitative objectives on the representation itself, such as disentanglement of latent variables BID12 or maximisation of mutual information BID3 BID0 BID8 .Mixup BID26 and manifold mixup BID21 are regularisation techniques that encourage deep neural networks to behave linearly between two data samples. These methods artificially augment the training set by producing random convex combinations between pairs of examples and their corresponding labels and training the network on these combinations. This has the effect of creating smoother decision boundaries, which can have a positive effect on generalisation performance. In BID21, the random convex combinations are computed in the hidden space of the network. This procedure can be viewed as using the high-level representation of the network to produce novel training examples and provides improvements over strong baselines in the supervised learning. Furthermore, BID22 propose a simple and efficient method for semi-supervised classification based on random convex combinations between unlabeled samples and their predicted labels. In this paper we explore the use of a wider class of mixing functions for unsupervised learning, mixing in the bottleneck layer of an autoencoder. These mixing functions could consist of continuous interpolations between latent vectors such as in BID21, to binary masking operations to even a deep neural network which learns the mixing operation. In order to ensure that the output of the decoder given the mixed representation resembles the data distribution at the pixel level, we leverage adversarial learning BID4, where here we train a discriminator to distinguish between decoded mixed and unmixed representations. This technique affords a model the ability to simulate novel data points (such as those corresponding to combinations of annotations not present in the training set). Furthermore, we explore our approach in the context of semi-supervised learning, where we learn a mixing function whose objective is to produce interpolations of hidden states consistent with a conditioned class label. Our method can be thought of as an extension of autoencoders that allows for sampling through mixing operations, such as continuous interpolations and masking operations. Variational autoencoders can also be thought of as a similar extension of autoencoders, using the outputs of the encoder as parameters for an approximate posterior q(z|x) which is matched to a prior distribution p(z) through the evidence lower bound objective (ELBO). At test time, new data points are sampled by passing samples from the prior, z ∼ p(z), through the decoder. In contrast, our we sample a random mixup operation between the representations of two inputs from the encoder. The Adversarially Constrained Autoencoder Interpolation (ACAI) method is another approach which involves sampling interpolations as part of an unsupervised objective BID2. ACAI uses a discriminator network to predict the mixing coefficient from the decoder output of the mixed representation, and the autoencoder tries to'fool' the discriminator, making interpolated points indistinguishable from real ones. The GAIA algorithm BID18 ) uses a BEGAN framework with an additional interpolation-based adversarial objective. What primarily differentiates our work from theirs is that we perform an exploration into different kinds of mixing functions, including a semi-supervised variant which uses an MLP to produce mixes consistent with a class label. Let us consider an autoencoder model F (·), with the encoder part denoted as f (·) and the decoder g(·). In an autoencoder we wish to minimise the reconstruction, which is simply: DISPLAYFORM0 Because autoencoders trained by input-reconstruction loss tend to produce images which are slightly blurry, one can train an adversarial autoencoder BID14, but instead of putting the adversary on the bottleneck, we put it on the reconstruction, and the discriminator (denoted D) tries to distinguish between real and reconstructed x, and the autoencoder (which is analogous to the generator) tries to construct'realistic' reconstructions so as to fool the discriminator. Because of this, we coin the term'ARAE' (adversarial reconstruction autoencoder). This can be written as: DISPLAYFORM1 where GAN is a GAN-specific loss function. In our case, GAN is the binary cross-entropy loss, which corresponds to the Jenson-Shannon GAN BID4.One way to use the autoencoder to generate novel samples would be to encode two inputs h 1 = f (x 1) and h 2 = f (x 2) into their latent representation, perform some combination between them, and then run the through the decoder g(·). There are many ways one could combine the two latent representations, and we denote this function Mix(h 1, h 2). Manifold mixup BID21 implements mixing in the hidden space through convex combinations: DISPLAYFORM2 where λ ∈ (bs,) is sampled from a Uniform distribution and bs denotes the minibatch size. In contrast, here we explore a strategy in which we randomly retain some components of the hidden representation from h 1 and use the rest from h 2, and in this case we would randomly sample a binary mask m ∈ {0, 1} (bs×f) (where f denotes the number of feature maps) and perform the following operation: DISPLAYFORM3 where m is sampled from a Bernoulli(p) distribution (p can simply be sampled uniformly).With this in mind, we propose the adversarial mixup resynthesiser (AMR), where part of the autoencoder's objective is to produce mixes which, when decoded, are indistinguishable from real images. The generator and the discriminator of AMR are trained by the following mixture of loss components: DISPLAYFORM4 fool D with reconstruction DISPLAYFORM5 fool D with mixes DISPLAYFORM6 label reconstruction as fake DISPLAYFORM7 label mixes as fake.Note that the mixing consistency loss is simply the reconstruction between the mixh mix = Mix(f (x), f (x)) and the re-encoding of it f (g(h mix)), where x and x are two randomly sampled images from the training set. This may be necessary as without it the decoder may simply output an image which is not semantically consistent with the two images which were mixed (refer to Section 5.2 for an in-depth explanation and analysis of this loss). Both the generator and discriminator are trained by the decoded image of the mix g(Mix(f (x), f (x))). The discriminator D is trained to label it as a fake image by minimising its probability and the generator F is trained to fool the discriminator by maximising its probability. Note that the coefficient λ controls the reconstruction and the coefficient β controls the mixing consistency in the generator. See Figure 1 for a visualisation of the AMR model. While it is interesting to generate new examples via random mixing strategies in the hidden states, we also explore a supervised mixing formulation in which we learn a mixing function that can produce mixes between two examples such that they are consistent with a particular class label. We make this possible by backpropagating through a classifier network p(y|x) which branches off the end of the discriminator, i.e., an auxiliary classifier GAN BID16.Let us assume that for some image x, we have a set of binary attributes y associated with it, where y ∈ {0, 1} k (and k ≥ 1). We introduce a mixing function Mix sup (h 1, h 2, y), which is an MLP that maps y to Bernoulli parameters p ∈ bs×f. These parameters are used to sample a Bernoulli mask m ∼ Bernoulli(p) to produce a new combinationh mix = mh 1 + (1 − m)h 2, which is consistent with the class label y. Note that the conditioning class label should be semantically meaningful with respect to both of the conditioned hidden states. For example, if we're producing mixes based on the gender attribute and both h 1 and h 2 are male, it would not make sense to condition on the'female' label. To enforce this constraint, we simply make the conditioning label a convex combinationỹ mix = αy 1 + (1 − α)y 2 as well, using α ∼ Uniform. DISPLAYFORM0 The unsupervised version of the adversarial mixup resynthesiser (AMR). In addition to the autoencoder loss functions, we have a mixing function Mix which creates some combination between the latent variables h 1 and h 2, which is subsequently decoded into an image intended to be realistic-looking and semantically consistent with the two constituent images. This is achieved through the consistency loss (weighted by β) and the discriminator. To make this more concrete, the autoencoder and discriminator, in addition to their losses described in Equation 5, try to minimise the following losses: DISPLAYFORM1 label mixes as fake DISPLAYFORM2 Note that for the consistency loss the same coefficient β is used. See Figure 2 for a visualisation of the supervised AMR model. We use ResNets BID6 for both the generator and discriminator. The precise architectures for generator and discriminator can be found here.1 The datasets evaluated on are:• UT Zappos50K BID24: a large dataset comprising 50k images of shoes, sandals, slippers, and boots. Each shoe is centered on a white and in the same orientation, which makes it convenient for generative modelling purposes.• CelebA BID13: a large-scale and highly diverse face dataset consisting of 200K images. We use the aligned and cropped version of the dataset downscaled to 64px, and only consider (via the use of a keypoint-based heuristic) frontal faces. It is worth noting that despite this, there is still quite a bit of variation in terms of the size and position of the faces, which can make mixing between faces a more difficult task since the faces are not completely aligned.mixing with labels mixing without labels Figure 2: The supervised version of the adversarial mixup resynthesiser (AMR). The mixer function, denoted in this figure as Mix sup, takes h 1, h 2 and a convex combination of y 1 and y 2 (denotedỹ mix) and internally produces a Bernoulli mask which is then used to produce an output combinatioñ h mix = mh 1 + (1 − m)h 2.h mix is then passed to the generator to generatex mix. In addition to fooling the discriminator usingx mix, the generator also has to make sure the class prediction by the auxiliary classifier is consistent with the mixed classỹ mix. Note that in this formulation, we still perform the kind of mixing which was shown in Figure 1, and this is shown in the diagram with the component noted'mixing without labels'. BID2 and (d) the adversarial mixup resynthesiser (AMR). For more images, consult the appendix section. As seen in FIG0, all of the mixup variants produce more realistic-looking interpolations than in pixel space. Due to details in CelebA however, it is slightly harder to distinguish the quality between the different methods. Though this may not be the most ideal metric to use in our case (see discussion at end of this section) we use the Frechet Inception Distance (FID) by BID7, which is based on features extracted from a pre-trained CelebA classifier, to compute the distance between samples from the dataset and ones from our autoencoders 2. Concretely, we compute (on the validation set) two scores: the FID between validation samples and their reconstructions (denoted in the table as FID(data, reconstruction)), and the FID between validation samples and randomly sampled interpolations (denoted in the table as FID(data, mix)). In the latter case, we repeat this five times (over five different sets of randomly sampled interpolations) for three different random seeds, ing in 5 × 3 = 15 FID scores from which we compute the mean and standard deviation. These are shown in TAB0 for the mixup and Bernoulli mixup formulations, respectively. Lower FID is usually considered to be better. However, FID may not be the most appropriate metric to use in our case. Because the FID is a measure of distance between two distributions, one can simply obtain a very low FID by simplying autoencoding the data, as shown in TAB0. In the case of mixing, one situation which may favour a lower FID is if g(αf ( DISPLAYFORM0 ; in other words, the supposed mix simply decodes into one of the original examples x 1 or x 2, which clearly lie on the data manifold. To avoid having the mixed features αx 1 + (1 − α)x 2 being decoded back into samples which lie on the data manifold, we leverage the consistency loss, which is tuned by coefficient β. The lower the coefficient, the more likely that decoded mixes are projected back onto the manifold, but if this constraint is too weak then it may not necessarily be desirable if one wants to create novel data points. (For more details, see Section 5.2 in the appendix.)Despite potential shortcomings of using FID, it seems reasonable to use such a metric to compare against baselines without any mixing losses, such as the adversarial reconstruction autoencoder (ARAE), which we indeed outperform for both mixup and Bernoulli mixup. For Bernoulli mixup, the FID scores appear to be higher than those in the mixup case TAB0 because the sampled Bernoulli mask m is also across the channel axis, i.e., it is of the shape (bs, f), whereas in mixup α has the shape (bs,). Because the mixing is performed on an extra axis, this produces a greater degree of variability in the mixes, and we have observed similar FID scores to the Bernoulli mixup case by evaluating on a variant of mixup where the α has the shape (bs, f) instead of (bs,). We present some qualitative with the supervised formulation. We train our supervised AMR variant using a subset of the attributes in CelebA ('is male', 'is wearing heavy makeup', and 'is wearing lipstick'). We consider pairs of examples {(x 1, y 1), (x 2, y 2)} (where one example is male and the other female) and produce random convex combinations of the attributesỹ mix = αy 1 + (1 − α)y 2 and decode their ing mixes Mix sup (f (x 1), f (x 2),ỹ mix ). This can be seen in FIG2.: Interpolations produced by the class mixer function for the set of binary attributes {male, heavy makeup, lipstick}. For each image, the left-most face is x 1 and the right-most face x 2, with faces in between consisting of mixes Mix sup (f (x 1), f (x 2),ỹ mix ) of a particular attribute mixỹ mix, shown below each column (where red denotes 'off' and green denotes 'on').We can see that for the most part, the class mixer function has been able to produce decent mixes between the two faces consistent with the desired attributes. There are some issues -namely, the model does not seem to disentangle the lipstick and makeup attributes well -but this may be due to the strong correlation between lipstick and makeup (lipstick is makeup!), or be in part due to the classification performance of the auxiliary classifier part of the discriminator (while its accuracy on both training and validation was as high as 95%, there may still be room for improvement). We also achieved better by simply having the embedding function produce a mask m ∈ rather than {0, 1}, most likely because such a formulation allows a greater degree of flexibility when it comes to mixing. Indeed, one priority is to conduct further hyperparameter tuning in order to improve these . For a visualisation of the Bernoulli parameters output by the embedding function, see Section 5.3 in the appendix. In this paper, we proposed the adversarial mixup resynthesiser and showed that it can be used to produce realistic-looking combinations of examples by performing mixing in the bottleneck of an autoencoder. We proposed several mixing functions, including one based on sampling from a uniform distribution and the other a Bernoulli distribution. Furthermore, we presented a semisupervised version of the Bernoulli variant in which one can leverage class labels to learn a mixing function which can determine what parts of the latent code should be mixed to produce an image consistent with a desired class label. While our technique can be used to leverage an autoencoder as a generative model, we conjecture that our technique may have positive effects on the latent representation and therefore downstream tasks, though this is yet to be substantiated. Future work will involve more comparisons to existing literature and experiments to determine the effects of mixing on the latent space itself and downstream tasks. We will provide a summary of our experimental setup here, though we also provide links to (and encourage viewers to look at) various parts of the code such as the networks used for the generator and discriminator and the optimiser hyperparameters. We use a residual network for both the generator and discriminator. The discriminator uses spectral normalisation BID15, with five discriminator updates being performed for each generator update. We use ADAM for our optimiser with α = 2e −4, β 1 = 0.0 and β 2 = 0.99. In order to examine the effect of the consistency loss, we explore a simple two-dimensional spiral dataset, where points along the spiral are deemed to be part of the data distribution and points outside it are not. With the mixup loss enabled and λ = 10, we try values of β ∈ {0, 0.1, 10, 100}. After 100 epochs of training, we produce decoded random mixes and plot them over the data distribution, which are shown as orange points (overlaid on top of real samples, shown in blue). This is shown in FIG3.As we can see, the lower β is, the more likely interpolated points will lie within the data manifold (i.e. the spiral). This is because the consistency loss competes with the discriminator loss -as β is decreased, there is a relatively greater incentive for the autoencoder to try and fool the discriminator with interpolations, forcing it to decode interpolated points such that they lie in the spiral. Ideally however we would want a bit of both: we want high consistency so that interpolations in hidden states are semantically meaningful (and do not decode into some other random data point), while also having those decoded interpolations look realistic. Interpolations are defined as DISPLAYFORM0 We also compare our formulation to ACAI BID2, which does not explicitly have a consistency loss term. Instead, the discriminator tries to predict what the mixing coefficient α is, and the autoencoder tries to fool it into thinking interpolations have a coefficient of 0. In FIG5 we compare this to our formulation in which β = 0. This is shown in FIG5 (right figure). It appears that ACAI also prefers to place points in the spiral, although not as strongly as AMR with β = 0 (though this may be because ACAI needs to trained for longer -ACAI and AMR were trained for the same number of epochs). In FIG5 we can see that over the course of training the consistency losses for both ACAI and AMR gradually rise, indicating both models' preference for moving interpolated points closer to the data manifold. Note that here we are only observing the consistency loss during training, and it is not used in the generator's loss. Lastly, in Figure 9 we show some side-by-side comparisons of our model interpolating between faces when β = 50 and β = 0. We can see that when β = 0 interpolations between faces are not as smooth in terms of colour and lighting. This somewhat slight discontinuity in the interpolation may be explained by the decoder pushing these interpolated points closer to the data manifold, since there is no consistency loss enforced.(a) Left: AMR with λ = 10, β = 0; right: ACAI with λ = 10 (β = 0 since ACAI does not enforce a consistency loss). AMR was trained for 200 epochs and ACAI for 300 epochs, since ACAI takes longer to converge. DISPLAYFORM1 and α ∼ U for randomly sampled {x 1, x 2}) over the course of training. DISPLAYFORM2 Figure 9: Interpolations using AMR {λ = 50, β = 50} and {λ = 50, β = 0}. To recap, the class mixer in the supervised formulation internally maps from a labelỹ mix to Bernoulli parameters p ∈ K, from which a Bernoulli mask m ∼ Bernoulli(p) is sampled. The ing Bernoulli parameters p are shown in Figure 10, where each row denotes some combination of attributes y ∈ {000, 001, 010, . . .} and the columns denote the index of p (spread out across four images, such that the first image denotes p 1:128, second image p 128:256, etc.). We can see that each attribute combination spells out a binary combination of feature maps, which allows one to easily glean which feature maps contribute to which attributes. Figure 10: Visualisation of Bernoulli parameters p internally produced by the class mixer function. Rows denote attribute combinations y and columns denote the index of p. In this section we show additional samples of the AMR model (using mixup and Bernoulli mixup variants) on Zappos and CelebA datasets. We compare AMR against linear interpolation in pixel space (pixel), adversarial reconstruction autoencoder (ARAE), and adversarialy contrained autoencoder interpolation (ACAI). As can be observed in the following images, the interpolations of pixel and ARAE are less realistic and suffer from more artifacts. AMR and ACAI produce more realisticlooking , while AMR generates a smoother transition between the two samples.• Figure

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.

Operations in the GAN latent space can induce a distribution mismatch compared to the training distribution, and we address this using optimal transport to match the distributions.


Question: Represent the scientific article to find a short TLDR!

Question: We outline the problem of concept drifts for time series data. In this work, we analyze the temporal inconsistency of streaming wireless signals in the context of device-free passive indoor localization. We show that data obtained from WiFi channel state information (CSI) can be used to train a robust system capable of performing room level localization. One of the most challenging issues for such a system is the movement of input data distribution to an unexplored space over time, which leads to an unwanted shift in the learned boundaries of the output space. In this work, we propose a phase and magnitude augmented feature space along with a standardization technique that is little affected by drifts. We show that this robust representation of the data yields better learning accuracy and requires less number of retraining. Concept drift is one of the most common problems that degrades the predictive performance of passive WiFi-based localization systems. In most of the predictive models it is assumed that a static relationship between input and output exits. Thus in the context of machine learning, there is a mapping function f (x) = y, where the algorithm tries to estimate the underlying relationship between the input x and the output y. The presence of concept drift means that the accuracy of the predictive models that is trained from historical data degrades over time due to evolving nature of the data. Hence, predictive models often needs to be retrained frequently with a new set of labelled data, which might be expensive to obtain. These pattern changes can be categorized based on their transition speed from one state to another into abrupt, or gradual drifts BID1. In either case, the deployed solution is expected to diagnose unintended changes automatically and adapt accordingly. The problem of concept drift in WiFi-based localization systems, was first mentioned in BID2, which presents a technology that utilizes only off-the-shelf WiFi-enabled devices such as access points, laptops, smart TV for passive sensing in the environment of interest. The authors have applied an online semi-supervised approach to automatically detect gradual shifts in the feature space and propose an adaptive learning strategy to regain the prediction accuracy. We aim to address the same problem without making any assumption about the drift type. In this work, we illustrate that from time to time, both sudden and gradual drifts, can occur to the streaming WiFi data, which often hinder the performance of the trained models when tested on the measurements. Majority of the existing WiFi-based indoor localization systems are device-based, where the user's location is determined by a WiFi-enabled target device that needs to be carried by the subject all the time BID9. Practical challenges of using device-based approaches, impose some restrictions and therefore, a device-free and passive solution is a promising line of research both for academia and industry. For example, (a; b; BID5, are some of the existing research where device free passive WiFi localization is used along with deep learning. In BID0, the authors address drifts and the inconsistency of WiFi fingerprints for stationary subjects. However, most of these researches and their experiments were performed in a very controlled environment and within a limited time frames. On the other hand, the effect of concept drift mostly appears over time due to real-world conditions such as natural WiFi channel or bandwidth switches, or when certain exogenous factor such as temperature and humidity changes. Therefore, the existing methods do not address them explicitly and the experimental does not reflect the performance of the model taken from measurements that are a few days apart. In this paper, we use the idea of feature augmentation in order to include both phase and magnitude of the CSI data. To the best of our knowledge this is the first work that exploits both the phase and magnitude of the CSI in order to construct a feature space that is less affected by drifts. We show that once such a feature space has been constructed,we can use classical machine learning algorithms in order to create a more robust model. In the next sections, we discuss nature of the WiFi CSI data being obtained and how drifts cause a shift in the feature space. In Section 3 we discuss our methods including the phase and the magnitude sanitization procedure. In Section ?? we present the training strategy for off line training and online prediction. Finally in Section 5, we conclude our paper and present discussions on future work. In wireless communication, channel state information (CSI) contains potential information that describes the propagation of a signal from transmitter to receiver. The CSI contains vital information that describes the combined effect of scattering, fading and decay with distance. In other words CSI reflects the variation in the channel that is experienced during propagation. Transmitted from a source, a wireless signal can experience various forms of distortion including fading, shadowing and multipath effects. For our application, we considered a WiFi channel at the 5GHz band which can be considered as a flat fading channel. Our network interface card (NIC) implements an OFDM system with 56 subcarriers, all of which can be read from CSI measurement. The receiver (Rx) and the transmitter (Tx) have 4 antennas each and in total our NIC establishes 16 links or streams altogether (one per Rx-Tx pair). The channel frequency response CSI i,k for subcarrier i and stream k is a complex number which is defined as: DISPLAYFORM0 where |CSI ik | and ∠CSI ik denote the magnitude and the phase response respectively. Let I be the total number of subcarriers, M be the total number of packets and S be the total number of streams. Therefore, our system produces 3 dimensional complex gain matrix with cardinality |I|×|M|×|S|.In the sequel, we show WiFi mesh variation for an empty capture of an indoor area and a capture containing walking inside a particular room of that area, obtained on different time stamps. Figure 2 shows a similar capture taken after 9 hours. These figures illustrate the effect of drift on the WiFi mesh, both for empty and walking captures. This change in distribution of the data along the feature space is what we refer to as concept drift. In the next section we formally discuss the methodology that we adapt in order to construct a new robust feature space that is less affected by drift. In this section, we discuss the methods that we followed starting from the sanitization of the raw CSI data. Then We proceed on discussing the ways in which we incorporated both the phase and magnitude of the CSI and justify why the combined feature space is little affected by drifts. We start our data processing with Received signal strength indicator (RSSI) drops filter. This filter looks at the successive packets and measures sudden peaks for RSSI values, these peaks can be of constructive interference from neighboring devices, multipath fading and temporal dynamics. The filter then discards the corresponding packets from the CSI. Since we have multiple subcarriers and streams which correspond to different links between the transmitter and the receiver, at each point in time, they can take values which are scaled to a wide range. Hence after discarding the packets based on RSSI corrections, we perform normalization of the CSI amplitudes to a predefined range. The L 2 norm of the CSI vector is then calculated for each of the CSI vectors in order to re-scale their values to the predefined range. In this section we outline the process by which we extract the phase information from the CSI and use it for feature augmentation. Prior research conducted with extraction of phase information from CSI reported an extensive amount of preprocessing being involved BID10. In BID9, the authors discuss the stability of phase for consecutive antennas for 5Ghz OFDM channel. Since, our NIC implements a 5GHz OFDM channel, we utilize the fact that phase difference between successive antennas are stable. We consider the phase difference between stream 1-2,stream 2-3 and stream 3-4 as they correspond to the links from a single transmitter to all 4 of the receiving antennas. A phase correction is then performed for the phases such that their values lie with in the range (−π, π). We then use a Hampel filter in order to remove the DC component of the phase information and to detrend the phase data. For our Hampel filter we use a large sliding window of 300 samples and with a small threshold of 0.01 in order to get the general trend of the data. Once the trend has been computed, it is then removed from the the phase difference information. Then, we further leverage Hampel filter with a smaller sliding window of 15 samples and a threshold of 0.01 in order to remove the high frequency noise from the streaming phase data. In Figure 3 we can see that the raw phase information has a wider spread and hence is more unstable compared to the phase difference information between successive antennas. In Figure 3 (a) we show the plot for the raw phase information that corresponds to subcarrier 1 and stream 1 and in Figure 3 (b) we show the plots for the phase difference between stream 1 and stream 2 which corresponds to the links from a single transmitter and two adjacent receiving antennas. In the next section we discuss about the feature augmentation and the standardization technique that we use in order to create a feature space that is robust to drifts. We propose an augmented feature space, comprising of both phase and magnitude of the WiFi CSI. For our feature space we consider CSI magnitude for 8 streams and phase difference data from the first four streams i,e we take the phase difference between stream 1, stream 2; stream 2, stream3 and stream 3,stream 4. We consider the first 800 packets for our data, therefore for 8 streams and all 56 subcarriers the cardinality of the magnitude M of the CSI is of the order |56| × |800| × |4| whereas for the phase information the data matrix has a cardinality of |56| × |800| × |3|. Our combined feature space F suitable for learning is a 2-D matrix comprising of both phase and magnitude with cardinality |800| × |392| for each location class (Room). Once the augmented feature space is obtained, we perform a standardization that standardizes the features by removing the mean and scaling to unit variance. Figure 4 shows the change is feature space observed when there is a drift, Fig 5 shows that the augmented feature space is almost resistant to drifts. In the next section we discuss broadly the of different learning algorithms for this augmented feature space. We discuss the training strategy that we have adapted for the experiments and present detailed and the effect of each algorithm on this enriched feature space. We present an offline training and an online prediction strategy for our system. We use classical machine learning algorithms to train on the un-drifted dataset using the augmented features. During testing, the algorithm in tested on the drifted data, which when projected to the combined feature space is least resistant to drifts. Thus trained models are used for the online prediction of the data that has drifts. We compare the performance of different learning algorithms for our training and classification for, a) the case of training only on magnitude data, b) training only on phase data and c) training on the combined data, which represents our most stable feature space. In this scenario, we train the data offline and perform classification using different learning algorithms. We mainly use Support Vector Machines (SVM) BID3 and Random Forest algorithms BID4. We then provide an incremental learning framework, that is popular with streaming data, specially for dealing with datasets associated with concept drifts. In the sequel we provide a detailed analysis of the performance of learning under these different frameworks and present the suitability of a learning framework that will be used for the real time localization application. We start our data collection procedure from apartments of three different sizes with different layouts (Apt 1, Apt 2 and Apt 3). We use two different devices for the experiments. Namely, the Tx and the Rx which corresponds to the routers for transmission and the reception, respectively. Both of the devices are placed further apart in the apartment and the experiment begins by taking an empty capture at the first instance. This empty capture corresponds to no motion at any of the rooms in the apartment. We next proceed towards capturing 1 minute data by walking in each of the rooms of the apartment, respectively, in order to obtain annotated data. The data is then collected and processed and converted to the augmented feature space as described in Section 3.1, 3.2 and 3.3, respectively. For Apt 1 we captured 5 rounds of data each of which is roughly 30 minutes apart. Although drift is more apparent for measurements taken over longer intervals, for measurements associated with Apt 1 we force a channel switched (abrupt drift) before collecting the 5th round by switching the devices off. This ensures that a drift has occurred since drifts are expected during a channel change. For Apt 2 we perform a more rigorous measurement and hence take 6 rounds of data, where we captured 3 rounds which are 12 hours apart and the last 2 rounds which are 2 days apart. For Apt 2 we take the measurements in such a diverse manner so that the effect of drift can be thoroughly studied. Finally for Apt 3, we capture 3 rounds of data where round 1 and round 2 are data which are 6 hours apart and round 3 is captured at an interval of 12 hours from round 2. FIG3 shows the layout of the three apartments in which the experiments were conducted. Through all of the experiments we ensured that the position of Tx and Rx remains fixed. Although our proposed augmented feature space in a dataset with high dimension, we found through repetitive experiments that using Principal Component Analysis (PCA) for dimensionality reduction yields very poor classification accuracy even when appropriate components are chosen based on explained cumulative variance analysis. Hence, we do not perform any dimensionality reduction on our dataset. For our experiments, all the learning algorithms are trained on rounds showing no drift and tested on rounds that has both gradual and abrupt drifts. In order to validate whether walking in different rooms of an indoor space actually correspond to distinguishable clusters from WiFi propagation perspective, we do an unsupervised clustering analysis over the dataset to evaluate our location partitioning. From the elbow analysis of the unsupervised clustering we found that WiFi mesh distortions can also be categorized in an unsupervised manner, where the number of clusters correspond to walking or physical activities in number of areas in the apartment. FIG4 presents the elbow analysis done in Apt 1 which consists of labelled data for 6 locations. For the elbow analysis described in FIG4 we can see that there is not much reduction in distortion when increasing the number of clusters from 6 to 8 thus we can conclude that the way in which we label the data in fact represents the different distribution that arises due to motion in different positions. In order to do offline training, we use Support Vector Machines (SVM) and Random Forest (RF) classifiers as the base learners. For each of these learners, we consider a K class classification problem where K is the number of positions / rooms where walking is performed. We chose SVM since it is effective in high dimensional spaces and because of its memory efficiency since it only uses a subset of the dataset in order to calculate the support vectors. We implement the SVM for performing non-linear classification using RBF kernels. We chose the Random Forest classifier, since it is a meta estimator that creates decision trees for different sub-samples of the training set and averaging the performance over them in order to find a better predictive accuracy. The performance for both the classifiers are compared when trained on the rounds with no drift and tested on the rounds with drift, which is set aside as a held out set. We also compare the time of training of these two base learners in order to justify the suitability of the corresponding learner for real-time indoor localization. In this section we present an incremental learning algorithm based on the proposed feature space. We chose an incremental learning framework since this allows the input data to continuously extend the existing knowledge of the model. Although the proposed feature space is almost resistant to concept drifts, we incorporate incremental learning so that the model can adapt to new data without forgetting its existing knowledge, in such a way it will adapt quickly to a very slow change in distribution of the data. For our learner, we use a SGD classifier with hinge loss and L2 regularizer, this in an SVM that can be updated incrementally. For the incremental learning framework, we keep on updating the model parameters for each round with the augmented feature space, where minimal drift is present. We then test the models on the rounds, where the feature space corresponding to only phase or the magnitude would perform poorly, and use our proposed stable feature space for the multi-class classification problem. TAB0 shows the performance of different learning algorithms for CSI features incorporating magnitude only, phase only and the proposed augmented feature space. The table shows that for all the learning algorithms the augmented feature space performs better and is more resistant to drifts. Also we note that in case of streaming data, the performance of incremental SVM described is consistently better for the augmented feature space. Thus we show that in case of large incoming data, the augmented feature space presents more robust features in terms of representing the WiFi CSI for localization. We have presented a comprehensive study in order to handle drifts for WiFi CSI data. We focused on the challenges presented by drifts for the application of indoor localization and proposed a combined feature space that is robust to drifts. We then incorporate this augmented feature space and provided a detailed analysis of the performance of different learning algorithms. Although we mainly focus on off line training, our work also focuses on robust online prediction in the presence of drifts. Such a stable feature space will will mean that we do not have to learn the abrupt and gradual drifts and retrain our models each time when there one. Our proposed feature space will also allow for applying deep convolution neural network, that has been only applied to either the phase or the magnitude information, but not both. The proposed feature space can be projected into an RGB image where, vital information can captured using a convolution layer which we keep for future work.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

We introduce an augmented robust feature space for streaming wifi data that is capable of tackling concept drift for indoor localization

represent the short summary for finding the corresponding scientific article (long)

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Federated learning improves data privacy and efficiency in machine learning performed over networks of distributed devices, such as mobile phones, IoT and wearable devices, etc. Yet models trained with federated learning can still fail to generalize to new devices due to the problem of domain shift. Domain shift occurs when the labeled data collected by source nodes statistically differs from the target node's unlabeled data. In this work, we present a principled approach to the problem of federated domain adaptation, which aims to align the representations learned among the different nodes with the data distribution of the target node. Our approach extends adversarial adaptation techniques to the constraints of the federated setting. In addition, we devise a dynamic attention mechanism and leverage feature disentanglement to enhance knowledge transfer. Empirically, we perform extensive experiments on several image and text classification tasks and show promising under unsupervised federated domain adaptation setting. Data generated by networks of mobile and IoT devices poses unique challenges for training machine learning models. Due to the growing storage/computational power of these devices and concerns about data privacy, it is increasingly attractive to keep data and computation locally on the device . Federated Learning (FL) (; ;) provides a privacy-preserving mechanism to leverage such decen-tralized data and computation resources to train machine learning models. The main idea behind federated learning is to have each node learn on its own local data and not share either the data or the model parameters. While federated learning promises better privacy and efficiency, existing methods ignore the fact that the data on each node are collected in a non-i.i.d manner, leading to domain shift between nodes . For example, one device may take photos mostly indoors, while another mostly outdoors. In this paper, we address the problem of transferring knowledge from the decentralized nodes to a new node with a different data domain, without requiring any additional supervision from the user. We define this novel problem Unsupervised Federated Domain Adaptation (UFDA), as illustrated in Figure 1 (a). There is a large body of existing work on unsupervised domain adaptation (; ; ; ;), but the federated setting presents several additional challenges. First, the data are stored locally and cannot be shared, which hampers mainstream domain adaptation methods as they need to access both the labeled source and unlabeled target data (; ; ; ; ;). Second, the model parameters are trained separately for each node and converge at different speeds, while also offering different contributions to the target node depending on how close the two domains are. Finally, the knowledge learned from source nodes is highly entangled , which can possibly lead to negative transfer . In this paper, we propose a solution to the above problems called Federated Adversarial Domain Adaptation (FADA) which aims to tackle domain shift in a federated learning system through adversarial techniques. Our approach preserves data privacy by training one model per source node and updating the target model with the aggregation of source gradients, but does so in a way that reduces domain shift. First, we analyze the federated domain adaptation problem from a theoretical perspective and provide a generalization bound. Inspired by our theoretical , we propose an Figure 1: (a) We propose an approach for the UFDA setting, where data are not shareable between different domains. In our approach, models are trained separately on each source domain and their gradients are aggregated with dynamic attention mechanism to update the target model. (b) Our FADA model learns to extract domain-invariant features using adversarial domain alignment (red lines) and a feature disentangler (blue lines). efficient adaptation algorithm based on adversarial adaptation and representation disentanglement applied to the federated setting. We also devise a dynamic attention model to cope with the varying convergence rates in the federated learning system. We conduct extensive experiments on real-world datasets, including image recognition and natural language tasks. Compared to baseline methods, we improve adaptation performance on all tasks, demonstrating the effectiveness of our devised model. Unsupervised Domain Adaptation Unsupervised Domain Adaptation (UDA) aims to transfer the knowledge learned from a labeled source domain to an unlabeled target domain. Domain adaptation approaches proposed over the past decade include discrepancy-based methods (; ; ; ;, reconstructionbased UDA models (; ;), and adversary-based approaches (; ; a;). For example, propose a gradient reversal layer to perform adversarial training to a domain discriminator, inspired by the idea of adversarial learning. address unsupervised domain adaptation by adapting a deep CNN-based feature extractor/classifier across source and target domains via adversarial training. introduce an H∆H-divergence to evaluate the domain shift and provide a generalization error bound for domain adaptation. These methods assume the data are centralized on one server, limiting their applicability to the distributed learning system. Federated Learning Federated learning (; ; ;) is a decentralized learning approach which enables multiple clients to collaboratively learn a machine learning model while keeping the training data and model parameters on local devices. Inspired by Homomorphic Encryption , gilad2016cryptonets propose CryptoNets to enhance the efficiency of data encryption, achieving higher federated learning performance. introduce a secure aggregation scheme to update the machine learning models under their federated learning framework. propose SecureML to support privacy-preserving collaborative training in a multi-client federated learning system. However, these methods mainly aim to learn a single global model across the data and have no convergence guarantee, which limits their ability to deal with non-i.i.d. data. To address the non-i.i.d data, introduce federated multi-task learning, which learns a separate model for each node. Liu et al. (2018b) propose semi-supervised federated transfer learning in a privacy-preserving setting. However, their models involve full or semi-supervision. The work proposed here is, to our best knowledge, the first federated learning framework to consider unsupervised domain adaptation. Feature Disentanglement Deep neural networks are known to extract features where multiple hidden factors are highly entangled. Learning disentangled representations can help remove irrelevant and domain-specific features and model only the relevant factors of data variation. To this end, recent work (; ; a;) explores the learning of interpretable representations using generative adversarial networks (GANs) and variational autoencoders (VAEs) . Under the fully supervised setting, propose an auxiliary classifier GAN (AC-GAN) to achieve representation disentanglement. (a) introduce a unified feature disentanglement framework to learn domain-invariant features from data across different domains. also extend VAEs into the semi-supervised setting for representation disentanglement. propose to disentangle the features into a domain-invariant content space and a domainspecific attributes space, producing diverse outputs without paired training data. Inspired by these works, we propose a method to disentangle the domain-invariant features from domain-specific features, using an adversarial training process. In addition, we propose to minimize the mutual information between the domain-invariant features and domain-specific features to enhance the feature disentanglement. We first define the notation and review a typical theoretical error bound for single-source domain adaptation devised by Ben-David et al. Then we describe our derived error bound for unsupervised federated domain adaptation. We mainly focus on the high-level interpretation of the error bound here and refer our readers to the appendix (see supplementary material) for proof details. Notation. Let D S 1 and D T denote source and target distribution on input space X and a ground-truth labeling function g: X → {0, 1}. A hypothesis is a function h: X → {0, 1} with the error w.r.t the ground-truth labeling function g: We denote the risk and empirical risk of hypothesis h on D S as S (h) and S (h). Similarly, the risk and empirical risk of h on D T are denoted as T (h) and T (h). The H-divergence between two distributions D and D is defined as: where H is a hypothesis class for input space X, and A H denotes the collection of subsets of X that are the support of some hypothesis in H. The symmetric difference space H∆H is defined as: H∆H:= {h(x) ⊕ h (x))|h, h ∈ H}, (⊕: the XOR operation). We denote the optimal hypothesis that achieves the minimum risk on the source and the target as h *:= arg min h∈H S (h) + T (h) and the error of h * as λ:= S (h *) + T (h *). Blitzer et al. (2007b) prove the following error bound on the target domain. Theorem 1. Let H be a hypothesis space of V C-dimension d and D S, D T be the empirical distribution induced by samples of size m drawn from D S and D T. Then with probability at least 1 − δ over the choice of samples, for each h ∈ H,, and be N source domains and the target domain in a UFDA system, where. In federated domain adaptation system, D S is distributed on N nodes and the data are not shareable with each other in the training process. The classical domain adaptation algorithms aim to minimize the target risk T (h):= Pr (x,y)∼D T [h(x) = y]. However, in a UFDA system, one model cannot directly get access to data stored on different nodes for security and privacy reasons. To address this issue, we propose to learn separate models for each distributed. The target hypothesis h T is the aggregation of the parameters of h S, i.e. We can then derive the following error bound: Theorem 2. (Weighted error bound for federated domain adaptation). Let H be a hypothesis class with VC-dimension d and, D T be empirical distributions induced by a sample of size m from each source domain and target domain in a federated learning system, respectively. Then, ∀α ∈ R N +, N i=1 α i = 1, with probability at least 1 − δ over the choice of samples, for each h ∈ H, 1 In this literature, the calligraphic D denotes data distribution, and italic D denotes domain discriminator. where λ i is the risk of the optimal hypothesis on the mixture of D Si and T, andS is the mixture of source samples with size N m. d H∆H (D Si, D T) denotes the divergence between domain S i and T. Comparison with Existing Bounds The bound in is extended from and they are equivalent if only one source domain exists (N = 1). provide a generalization bound for multiple-source domain adaptation, assuming that the target domain is a mixture of the N source domains. In contrast, in our error bound, the target domain is assumed to be an novel domain, ing in a bound involving H∆H discrepancy and the VC-dimensional constraint . Blitzer et al. (2007b) propose a generalization bound for semisupervised multi-source domain adaptation, assuming that partial target labels are available. Our generalization bound is devised for unsupervised learning. introduce classification and regression error bounds for multi-source domain adaptation. However, these error bounds assume that the multiple source and target domains exactly share the same hypothesis. In contrast, our error bound involves multiple hypotheses. The error bound in Theorem demonstrates the importance of the weight α and the discrepancy Inspired by this, we propose dynamic attention model to learn the weight α and federated adversarial alignment to minimize the discrepancy between the source and target domains, as shown in Figure 1. In addition, we leverage representation disentanglement to extract domain-invariant representations to further enhance knowledge transfer. Dynamic Attention In a federated domain adaptation system, the models on different nodes have different convergence rates. In addition, the domain shifts between the source domains and target domain are different, leading to a phenomenon where some nodes may have no contribution or even negative transfer to the target domain. To address this issue, we propose dynamic attention, which is a mask on the gradients from source domains. The philosophy behind the dynamic attention is to increase the weight of those nodes whose gradients are beneficial to the target domain and limit the weight of those whose gradients are detrimental to the target domain. Specifically, we leverage the gap statistics to evaluate how well the target features f t can be clustered with unsupervised clustering algorithms (K-Means). Assuming we have k clusters, the gap statistics are computed as: where we have clusters C 1, C 2,..., C k, with C r denoting the indices of observations in cluster r, and n r =|C r |. Intuitively, a smaller gap statistics value indicates the feature distribution has smaller intraclass variance. We measure the contribution of each source domain by the gap statistics gain between two consecutive iterations: (p indicating training step), denoting how much the clusters can be improved before and after the target model is updated with the i-th source model's gradient. The mask on the gradients from source domains is defined as Sof tmax(I Federated Adversarial Alignment The performance of machine learning models degrades rapidly with the presence of domain discrepancy . To address this issue, existing work proposes to minimize the discrepancy with an adversarial training process. For example, proposes the domain confusion objective, under which the feature extractor is trained with a cross-entropy loss against a uniform distribution. However, these models require access to the source and target data simultaneously, which is prohibitive in UFDA. In the federated setting, we have multiple source domains and the data are locally stored in a privacy-preserving manner, which means we cannot train a single model which has access to the source domain and target domain simultaneously. To address this issue, we propose federated adversarial alignment that divides optimization into two independent steps, a domain-specific local feature extractor and a global discriminator. Specifically, for each domain, we train a local feature extractor, G i for D i and G t for D t, for each (D i, D t) source-target domain pair, we train an adversarial domain identifier DI to align the distributions in an adversarial manner: we first train DI to identify which domain are the features come from, then we train the generator (G i, G t) to confuse the DI. Note that D only gets access to the output vectors of G i and G t, without violating the UFDA Sample mini-batch from from {(x Compute gradient with cross-entropy classification loss, update Update with Eq. 4 and Eq. 5 respectively to align the domain distribution; 7: Domain Disentangle: 8: update Update .. Θ C N} respectively with the computed dynamic weight; 17: end while setting. Given the i-th source domain data X Si, target data X T, the objective for DI i is defined as follows: In the second step, L adv D remains unchanged, but L adv G is updated with the following objective: Representation Disentanglement We employ adversarial disentanglement to extract the domaininvariant features. The high-level intuition is to disentangle the features extracted by (G i, G t) into domain-invariant and domain-specific features. As shown in Figure 1(b), the disentangler D i separates the extracted features into two branches. Specifically, we first train the K-way classifier C i and K-way class identifier CI i to correctly predict the labels with a cross-entropy loss, based on f di and f ds features, respectively. The objective is: where ) denote the domain-invariant and domain-specific features respectively. In the next step, we freeze the class identifier CI i and only train the feature disentangler to confuse the class identifier CI i by generating the domain-specific features f ds, as shown in Figure 1. This can be achieved by minimizing the negative entropy loss of the predicted class distribution. The objective is as follows: Feature disentanglement facilitates the knowledge transfer by reserving f di and dispelling f ds. To enhance the disentanglement, we minimize the mutual information between domain-invariant features and domain-specific features, following. Specifically, the mutual information is The most gorgeous artwork in comic books. It contains the most extraordinary and finest artwork of Alex Ross. In my opinion it is the best American animated film ever released. It has a beautiful story with a ton of laughs, a lot of teachable moments. My advice is if you need a CD rack that holds a lot of CD's? Save your money and invest in something nicer and more sturdy. I absolutely love this product. my neighbor has four little yippers and my hepard/chow mix was antogonized on our side of the fence. defined as I(f di ; f ds) = P×Q log dP PQ dP P ⊗P Q dP PQ, where P PQ is the joint probability distribution of (f di, f ds), and P P = Q dP PQ, P Q = Q dP PQ are the marginals. Despite being a pivotal measure across different distributions, the mutual information is only tractable for discrete variables, for a limited family of problems where the probability distributions are unknown . , we adopt the Mutual Information Neural Estimator (MINE) to estimate the mutual information by leveraging a neural network. Practically, MINE can be calculated p,q,θ) ). To avoid computing the integrals, we leverage Monte-Carlo integration to calculate the estimation: where (p, q) are sampled from the joint distribution, q is sampled from the marginal distribution, and T (p, q, θ) is the neural network parameteralized by θ to estimate the mutual information between P and Q, we refer the reader to MINE for implementation details. The domain-invariant and domain-specific features are forwarded to a reconstructor with a L2 loss to reconstruct the original features, aming to keep the representation integrity, as shown in Figure 1(b). The balance of the L2 reconstruction and mutual information can be achieved by adjusting the hyper-parameters of the L2 loss and mutual information loss. Optimization Our model is trained in an end-to-end fashion. We train federated alignment and representation disentanglement component with Stochastic Gradient Descent . The federated adversarial alignment loss and representation disentanglement loss are minimized together with the task loss. The detailed training procedure is presented in Algorithm 1. We test our model on the following tasks: digit classification (Digit-Five), object recognition (OfficeCaltech10 , DomainNet ) and sentiment analysis (Amazon Review dataset (a) ). Figure 2 shows some data samples and Table 9 (see supplementary material) shows the number of data per domain we used in our experiments. We perform our experiments on a 10 Titan-Xp GPU cluster and simulate the federated system on a single machine (as the data communication is not the main focus of this paper). Our model is implemented with PyTorch. We repeat every experiment 10 times on the Digit-Five and Amazon Review datasets, and 5 times on the Office-Caltech10 and DomainNet datasets, reporting the mean and standard derivation of accuracy. To better explore the effectiveness of different components of our model, we propose three different ablations, i.e. model I: with dynamic attention; model II: I + adversarial alignment; and model III: II + representation disentanglement. and the rest as the distributed source domains, leading to five transfer tasks. The detailed architecture of our model can be found in Table 7 (see supplementary material). Since many DA models (; ;) require access to data from different domains, it is infeasible to directly compare our model to these baselines. Instead, we compare our model to the following popular domain adaptation baselines: Domain Adversarial Neural Network (DANN) , Deep Adaptation Network (DAN) , Automatic DomaIn Alignment Layers (AutoDIAL) , and Adaptive Batch Normalization (AdaBN). Specifically, DANN minimizes the domain gap between source domain and target domain with a gradient reversal layer. DAN applies multi-kernel MMD loss to align the source domain with the target domain in Reproducing Kernel Hilbert Space. AutoDIAL introduces domain alignment layer to deep models to match the source and target feature distributions to a reference one. AdaBN applies Batch Normalization layer to facilitate the knowledge transfer between the source and target domains. When conducting the baseline experiments, we use the code provided by the authors and modify the original settings to fit federated DA setting (i.e. each domain has its own model), denoted by f -DAN and f -DANN. In addition, to demonstrate the difficulty of UFDA where accessing all source data with a single model is prohibative, we also perform the corresponding multi-source DA experiments (shared source data). The experimental are shown in To dive deeper into the feature representation of our model versus other baselines, we plot in Figure 3 (a)-3(d) the t-SNE embeddings of the feature representations learned on mm,mt,sv,sy→up task with source-only features, f -DANN features, f -DAN features and FADA features, respectively. We observe that the feature embeddings of our model have smaller intra-class variance and larger inter-class variance than f -DANN and f -DAN, demonstrating that our model is capable of generating the desired feature embedding and can extract domain-invariant features across different domains. Office-Caltech10 This dataset contains 10 common categories shared by Office-31 and Caltech-256 datasets . It contains four domains: Caltech (C), which are sampled from Caltech-256 dataset, Amazon (A), which contains images collected from amazon.com, Webcam (W) and DSLR (D), which contains images taken by web camera and DSLR camera under office environment. We leverage two popular networks as the backbone of feature generator G, i.e. AlexNet and ResNet . Both the networks are pre-trained on ImageNet . Other components of our model are randomly initialized with the normal distribution. In the learning process, we set the learning rate of randomly initialized parameters to ten times of the pre-trained parameters as it will take more time for those parameters to converge. Details of our model are listed in Table 9 (supplementary material). The experimental on Office-Caltech10 datasets are shown in Table 2. We utilize the same backbones as the baselines and separately show the . We make the following observations from the : Our model achieves 86.5% accuracy with an AlexNet Table 3: Accuracy (%) on the DomainNet dataset dataset under UFDA protocol. The upper table shows the based on AlexNet backbone and the table below are the based on ResNet Table 4: Accuracy (%) on "Amazon Review" dataset with unsupervised federated domain adaptation protocol. backbone and 87.1% accuracy with a ResNet backbone, outperforming the compared baselines., we calculate the approximate A-distanced A = 2 (1 − 2) for C,D,W→A and A,C,W→D tasks, where is the generalization error of a two-sample classifier (e.g. kernel SVM) trained on the binary problem of distinguishing input samples between the source and target domains. In Figure 4 (a), we plot for tasks with raw ResNet features, f -DANN features, and FADA features, respectively. We observe that thed A on DADA features are smaller than ResNet features and f -DANN features, demonstrating that FADA features are harder to be distinguished between source and target. To show how the dynamic attention mechanism benefits the training process, we plot the training loss w/ or w/o dynamic weights for A,C,W→D task in Figure 4 (b). The figure shows the target model's training error is much smaller when dynamic attention is applied, which is consistent with the quantitative . In addition, in A,C,W→D setting, the weight of A decreases to the lower bound after first a few epochs and the weight of W increases during the training process, as photos in both D and W are taken in the same environment with different cameras. To better analyze the error mode, we plot the confusion matrices for f -DAN and FADA on A,C,D->W task in Figure 4 (c)-4(d). The figures show that f -DAN mainly confuses "calculator" vs. "keyboard", "backpack" with "headphones", while FADA is able to distinguish them with disentangled features. Table 5: The ablation study show that the dynamic attention module is essential for our model. (rel, photos and real world images; and Sketch (skt), sketches of specific objects. This dataset is very large-scale and contains rich and informative vision cues across different domains, providing a good testbed for unsupervised federated domain adaptation. Some sample images can be found in Figure 2. Results The experimental on DomainNet are shown in Table 3. Our model achieves 28.9% and 30.3% accuracy with AlexNet and ResNet backbone, respectively. In both scenarios, our model outperforms the baselines, demonstrating the effectiveness of our model on large-scale dataset. Note that this dataset contains about 0.6 million images, and so even a one-percent performance improvement is not trivial. From the experiment , we can observe that all the models deliver less desirable performance when infograph and quickdraw are selected as the target domains. This phenomenon is mainly caused by the large domain shift between inf /qdr domain and other domains. Amazon Review (a) This dataset provides a testbed for cross-domain sentimental analysis of text. The task is to identify whether the sentiment of the reviews is positive or negative. The dataset contains reviews from amazon.com users for four popular merchandise categories: Books (B), DVDs (D), Electronics (E), and Kitchen appliances (K). , we utilize 400-dimensional bag-of-words representation and leverage a fully connected deep neural network as the backbone. The detailed architecture of our model can be found in Table 8 (supplementary materials). The experimental on Amazon Review dataset are shown in Table 4. Our model achieves an accuracy of 78.9% and outperforms the compared baselines. We make two major observations from the : Our model is not only effective on vision tasks but also performs well on linguistic tasks under UFDA learning schema. From the of model I and II, we can observe the dynamic attention and federated adversarial alignment are beneficial to improve the performance. However, the performance boost from Model II to Model III is limited. This phenomenon shows that the linguistic features are harder to disentangle comparing to visual features. To demonstrate the effectiveness of dynamic attention, we perform the ablation study analysis. The Table 5 shows the on "Digit-Five", Office-Caltech10 and Amazon Review benchmark. We observe that the performance drops in most of the experiments when dynamic attention model is not applied. The dynamic attention model is devised to cope with the varying convergence rates in the federated learning system, i.e., different source domains have their own convergence rate. In addition, it will increase the weight of a specific domain when the domain shift between that domain and the target domain is small, and decrease the weight otherwise. In this paper, we first proposed a novel unsupervised federated domain adaptation (UFDA) problem and derived a theoretical generalization bound for UFDA. Inspired by the theoretical , we proposed a novel model called Federated Adversarial Domain Adaptation (FADA) to transfer the knowledge learned from distributed source domains to an unlabeled target domain with a novel dynamic attention schema. Empirically, we showed that feature disentanglement boosts the performance of FADA in UFDA tasks. An extensive empirical evaluation on UFDA vision and linguistic benchmarks demonstrated the efficacy of FADA against several domain adaptation baselines. We provide the explanations of notations occurred in this paper. Table 6: Notations occurred in the paper. We provide the detailed model architecture (Table 7 and Table 7 : Model architecture for digit recognition task ("Digit-Five" dataset). For each convolution layer, we list the input dimension, output dimension, kernel size, stride, and padding. For the fullyconnected layer, we provide the input and output dimensions. For drop-out layers, we provide the probability of an element to be zeroed. We provide the detailed information of datasets. For Digit-Five and DomainNet, we provide the train/test split for each domain. For Office-Caltech10, we provide the number of images in each domain. For Amazon review dataset, we show the detailed number of positive reviews and negative reviews for each merchandise category. Table 9: Model architecture for image recognition task (Office-Caltech10 and DomainNet ). For each convolution layer, we list the input dimension, output dimension, kernel size, stride, and padding. For the fully-connected layer, we provide the input and output dimensions. For drop-out layers, we provide the probability of an element to be zeroed. Remark. The equation in Theorem 2 provides a theoretical error bound for unsupervised federated domain adaptation as it assumes that the source data distributed on different nodes can form a mixture source domain. In fact, the data on different node can not be shared under the federated learning schema. The theoretical error bound is only valid when the weights of models on all the nodes are fully synchronized.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

we present a principled approach to the problem of federated domain adaptation, which aims to align the representations learned among the different nodes with the data distribution of the target node.

represent this text to find a much much longer article about it with say hundreds-thousands of words

A method for learning better representations, that acts as a regularizer and despite its no significant additional computation cost , achieves improvements over strong baselines on Supervised and Semi-supervised Learning tasks.