Question: Represent the scientific article to find a short TLDR

Question: Modern deep neural networks can achieve high accuracy when the training distribution and test distribution are identically distributed, but this assumption is frequently violated in practice. When the train and test distributions are mismatched, accuracy can plummet. Currently there are few techniques that improve robustness to unforeseen data shifts encountered during deployment. In this work, we propose a technique to improve the robustness and uncertainty estimates of image classifiers. We propose AugMix, a data processing technique that is simple to implement, adds limited computational overhead, and helps models withstand unforeseen corruptions. AugMix significantly improves robustness and uncertainty measures on challenging image classification benchmarks, closing the gap between previous methods and the best possible performance in some cases by more than half. Current machine learning models depend on the ability of training data to faithfully represent the data encountered during deployment. In practice, data distributions evolve , models encounter new scenarios , and data curation procedures may capture only a narrow slice of the underlying data distribution . Mismatches between the train and test data are commonplace, yet the study of this problem is not. As it stands, models do not robustly generalize across shifts in the data distribution. If models could identify when they are likely to be mistaken, or estimate uncertainty accurately, then the impact of such fragility might be ameliorated. Unfortunately, modern models already produce overconfident predictions when the training examples are independent and identically distributed to the test distribution. This overconfidence and miscalibration is greatly exacerbated by mismatched training and testing distributions. Small corruptions to the data distribution are enough to subvert existing classifiers, and techniques to improve corruption robustness remain few in number. show that classification error of modern models rises from 22% on the usual ImageNet test set to 64% on ImageNet-C, a test set consisting of various corruptions applied to ImageNet test images. Even methods which aim to explicitly quantify uncertainty, such as probabilistic and Bayesian neural networks, struggle under data shift, as recently demonstrated by. Improving performance in this setting has been difficult. One reason is that training against corruptions only encourages networks to memorize the specific corruptions seen during training and leaves models unable to generalize to new corruptions . Further, networks trained on translation augmentations remain highly sensitive to images shifted by a single pixel . Others have proposed aggressive data augmentation schemes , though at the cost of a computational increase. demonstrates that many techniques may improve clean accuracy at the cost of robustness while many techniques which improve robustness harm uncertainty, and contrariwise. In all, existing techniques have considerable trade-offs. In this work, we propose a technique to improve both the robustness and uncertainty estimates of classifiers under data shift. We propose AUGMIX, a method which simultaneously achieves new state-of-the-art for robustness and uncertainty estimation while maintaining or improving accuracy on standard benchmark datasets. AUGMIX utilizes stochasticity and diverse augmentations, a Jensen-Shannon Divergence consistency loss, and a formulation to mix multiple augmented images to achieve state-of-the-art performance. On CIFAR-10 and CIFAR-100, our method roughly halves the corruption robustness error of standard training procedures from 28.4% to 12.4% and 54.3% to 37.8% error, respectively. On ImageNet, AUGMIX also achieves state-of-the-art corruption robustness and decreases perturbation instability from 57.2% to 37.4%. Code is available at https://github.com/google-research/augmix. Figure 2: Example ImageNet-C corruptions. These corruptions are encountered only at test time and not during training. Robustness under Data Shift. show that training against distortions can often fail to generalize to unseen distortions, as networks have a tendency to memorize properties of the specific training distortion. show training with various blur augmentations can fail to generalize to unseen blurs or blurs with different parameter settings. propose measuring generalization to unseen corruptions and provide benchmarks for doing so. construct an adversarial version of the aforementioned benchmark.; argue that robustness to data shift is a pressing problem which greatly affects the reliability of real-world machine learning systems. Calibration under Data Shift.; propose metrics for determining the calibration of machine learning models. find that simply ensembling classifier predictions improves prediction calibration. Hendrycks et al. (2019a) show that pre-training can also improve calibration. demonstrate that model calibration substantially deteriorates under data shift. Data Augmentation. Data augmentation can greatly improve generalization performance. For image data, random left-right flipping and cropping are commonly used. Random occlusion techniques such as Cutout can also improve accuracy on clean data . Rather than occluding a portion of an image, CutMix replaces a portion of an image with a portion of a different image. Mixup also uses information from two images. Rather than implanting one portion of an image inside another, Mixup produces an elementwise convex combination of two images (; Tokozume et al., Figure 3 : A cascade of successive compositions can produce images which drift far from the original image, and lead to unrealistic images. However, this divergence can be balanced by controlling the number of steps. To increase variety, we generate multiple augmented images and mix them. 2018). show that Mixup can be improved with an adaptive mixing policy, so as to prevent manifold intrusion. Separate from these approaches are learned augmentation methods such as AutoAugment , where a group of augmentations is tuned to optimize performance on a downstream task. Patch Gaussian augments data with Gaussian noise applied to a randomly chosen portion of an image. A popular way to make networks robust to p adversarial examples is with adversarial training , which we use in this paper. However, this tends to increase training time by an order of magnitude and substantially degrades accuracy on non-adversarial images . AUGMIX is a data augmentation technique which improves model robustness and uncertainty estimates, and slots in easily to existing training pipelines. At a high level, AugMix is characterized by its utilization of simple augmentation operations in concert with a consistency loss. These augmentation operations are sampled stochastically and layered to produce a high diversity of augmented images. We then enforce a consistent embedding by the classifier across diverse augmentations of the same input image through the use of Jensen-Shannon divergence as a consistency loss. Mixing augmentations allows us to generate diverse transformations, which are important for inducing robustness, as a common failure mode of deep models in the arena of corruption robustness is the memorization of fixed augmentations . Previous methods have attempted to increase diversity by directly composing augmentation primitives in a chain, but this can cause the image to quickly degrade and drift off the data manifold, as depicted in Figure 3. Such image degradation can be mitigated and the augmentation diversity can be maintained by mixing together the of several augmentation chains in convex combinations. A concrete account of the algorithm is given in the pseudocode below. x aug += w i · chain(x orig) Addition is elementwise Sample weight m ∼ Beta(α, α) Interpolate with rule x augmix = mx orig + (1 − m)x aug 13: return x augmix 14: end function 15: x augmix1 = AugmentAndMix(x orig) x augmix1 is stochastically generated 16: Augmentations. Our method consists of mixing the from augmentation chains or compositions of augmentation operations. We use operations from AutoAugment. Each operation is visualized in Appendix C. Crucially, we exclude operations which overlap with ImageNet-C corruptions. In particular, we remove the contrast, color, brightness, sharpness, and Cutout operations so that our set of operations and the ImageNet-C corruptions are disjoint. In turn, we do not use any image noising nor image blurring operations so that ImageNet-C corruptions are encountered only at test time. Operations such as rotate can be realized with varying severities, like 2 • or −15 •. For operations with varying severities, we uniformly sample the severity upon each application. Next, we randomly sample k augmentation chains, where k = 3 by default. Each augmentation chain is constructed by composing from one to three randomly selected augmentation operations. Mixing. The ing images from these augmentation chains are combined by mixing. While we considered mixing by alpha compositing, we chose to use elementwise convex combinations for simplicity. The k-dimensional vector of convex coefficients is randomly sampled from a Dirichlet(α, . . ., α) distribution. Once these images are mixed, we use a "skip connection" to combine the of the augmentation chain and the original image through a second random convex combination sampled from a Beta(α, α) distribution. The final image incorporates several sources of randomness from the choice of operations, the severity of these operations, the lengths of the augmentation chains, and the mixing weights. Jensen-Shannon Divergence Consistency Loss. We couple with this augmentation scheme a loss that enforces smoother neural network responses. Since the semantic content of an image is approximately preserved with AUGMIX, we should like the model to embed x orig, x augmix1, x augmix2 similarly. Toward this end, we minimize the Jensen-Shannon divergence among the posterior distributions of the original sample x orig and its augmented variants. That is, for p orig =p(y | x orig), p augmix1 =p(y | x augmix1), p augmix2 =p(y|x augmix2), we replace the original loss L with the loss To interpret this loss, imagine a sample from one of the three distributions p orig, p augmix1, p augmix2. The Jensen-Shannon divergence can be understood to measure the average information that the sample reveals about the identity of the distribution from which it was sampled. This loss can be computed by first obtaining M = (p orig + p augmix1 + p augmix2)/3 and then computing Unlike an arbitrary KL Divergence between p orig and p augmix, the Jensen-Shannon divergence is upper bounded, in this case by the logarithm of the number of classes. Note that we could instead compute JS(p orig ; p augmix1), though this does not perform as well. The gain of training with JS(p orig ; p augmix1 ; p augmix2 ; p augmix3) is marginal. The Jensen-Shannon Consistency Loss impels to model to be stable, consistent, and insensitive across to a diverse range of inputs (; ; . Ablations are in Section 4.3 and Appendix A. Datasets. The two CIFAR datasets contain small 32 × 32 × 3 color natural images, both with 50,000 training images and 10,000 testing images. CIFAR-10 has 10 categories, and CIFAR-100 has 100. The ImageNet dataset contains 1,000 classes of approximately 1.2 million large-scale color images. In order to measure a model's resilience to data shift, we evaluate on the CIFAR-10-C, CIFAR-100-C, and ImageNet-C datasets . These datasets are constructed by corrupting the original CIFAR and ImageNet test sets. For each dataset, there are a total of 15 noise, blur, weather, and digital corruption types, each appearing at 5 severity levels or intensities. Since these datasets are used to measure network behavior under data shift, we take care not to introduce these 15 corruptions into the training procedure. The CIFAR-10-P, CIFAR-100-P, and ImageNet-P datasets also modify the original CIFAR and ImageNet datasets. These datasets contain smaller perturbations than CIFAR-C and are used to measure the classifier's prediction stability. Each example in these datasets is a video. For instance, a video with the brightness perturbation shows an image getting progressively brighter over time. We should like the network not to give inconsistent or volatile predictions between frames of the video as the brightness increases. Thus these datasets enable the measurement of the "jaggedness" of a network's prediction stream. Metrics. The Clean Error is the usual classification error on the clean or uncorrupted test data. In our experiments, corrupted test data appears at five different intensities or severity levels 1 ≤ s ≤ 5. For a given corruption c, the error rate at corruption severity s is E c,s. We can compute the average error across these severities to create the unnormalized corruption error uCE c = 5 s=1 E c,s. On CIFAR-10-C and CIFAR-100-C we average these values over all 15 corruptions. Meanwhile, on ImageNet we follow the convention of normalizing the corruption error by the corruption error of AlexNet . We compute CE c =. The average of the 15 corruption errors CE Gaussian Noise, CE Shot Noise,..., CE Pixelate, CE JPEG gives us the Mean Corruption Error (mCE). Perturbation robustness is not measured by accuracy but whether video frame predictions match. Consequently we compute what is called the flip probability. Concretely, for videos such as those with steadily increasing brightness, we determine the probability that two adjacent frames, or two frames with slightly different brightness levels, have "flipped" or mismatched predictions. There are 10 different perturbation types, and the mean across these is the mean Flip Probability (mFP). As with ImageNet-C, we can normalize by AlexNet's flip probabilities and obtain the mean Flip Rate (mFR). In order to assess a model's uncertainty estimates, we measure its miscalibration. Classifiers capable of reliably forecasting their accuracy are considered "calibrated." For instance, a calibrated classifier should be correct 70% of the time on examples to which it assigns 70% confidence. Let the classifier's confidence that its predictionŶ is correct be written C. Then the idealized RMS Calibration Error is, which is the squared difference between the accuracy at a given confidence level and actual the confidence level. In Appendix E, we show how to empirically estimate this quantity and calculate the Brier Score. Training Setup. In the following experiments we show that AUGMIX endows robustness to various architectures including an All Convolutional Network , a DenseNet-BC (k = 12, d = 100) , a 40-2 Wide ResNet , and a ResNeXt-29 (32 × 4) . All networks use an initial learning rate of 0.1 which decays following a cosine learning rate . All input images are pre-processed with standard random left-right flipping and cropping prior to any augmentations. We do not change AUGMIX parameters across CIFAR-10 and CIFAR-100 experiments for consistency. The All Convolutional Network and Wide ResNet train for 100 epochs, and the DenseNet and ResNeXt require 200 epochs for convergence. We optimize with stochastic gradient descent using Nesterov momentum. Table 1: Average classification error as percentages. Across several architectures, AUGMIX obtains CIFAR-10-C and CIFAR-100-C corruption robustness that exceeds the previous state of the art. Results. Simply mixing random augmentations and using the Jensen-Shannon loss substantially improves robustness and uncertainty estimates. Compared to the "Standard" data augmentation baseline ResNeXt on CIFAR-10-C, AUGMIX achieves 16.6% lower absolute corruption error as shown in Figure 5. In addition to surpassing numerous other data augmentation techniques, Table 1 demonstrates that these gains directly transfer across architectures and on CIFAR-100-C with zero additional tuning. Crucially, the robustness gains do not only exist when measured in aggregate. Figure 12 shows that AUGMIX improves corruption robustness across every individual corruption and severity level. Our method additionally achieves the lowest mFP on CIFAR-10-P across three different models all while maintaining accuracy on clean CIFAR-10, as shown in Figure 6 (left) and Table 6. Finally, we demonstrate that AUGMIX improves the RMS calibration error on CIFAR-10 and CIFAR-10-C, as shown in Figure 6 (right) and Table 5. Expanded CIFAR-10-P and calibration are in Appendix D, and Fourier Sensitivity analysis is in Appendix B. Baselines. To demonstrate the utility of AUGMIX on ImageNet, we compare to many techniques designed for large-scale images. While techniques such as Cutout have not been demonstrated to help on the ImageNet scale, and while few have had success training adversarially robust models on ImageNet , other techniques such as Stylized ImageNet have been demonstrated to help on ImageNet-C. Patch Uniform ) is similar to Cutout except that randomly chosen regions of the image are injected with uniform noise; the original paper uses Gaussian noise, but that appears in the ImageNet-C test set so we use uniform noise. We tune Patch Uniform over 30 hyperparameter settings. Next, AutoAugment searches over data augmentation policies to find a high-performing data augmentation policy. We denote AutoAugment with AutoAugment* since we remove augmentation operations that overlap with ImageNet-C corruptions, as with AUGMIX. We also test with Random AutoAugment*, an augmentation scheme where each image has a randomly sampled augmentation policy using AutoAugment* operations. In contrast to AutoAugment, Random AutoAugment* and AUGMIX require far less computation and provide more augmentation variety, which can offset their lack of optimization. Note that Random AutoAugment* is different from RandAugment introduced recently by: RandAugment uses AutoAugment operations and optimizes a single distortion magnitude hyperparameter for all operations, while Random AutoAugment* randomly samples magnitudes for each operation and uses the same operations as AUGMIX. MaxBlur Pooling is a recently proposed architectural modification which smooths the of pooling. Now, Stylized ImageNet (SIN) is a technique where models are trained with the original ImageNet images and also ImageNet images with style transfer applied. Whereas the original Stylized ImageNet technique pretrains on ImageNet-C and performs style transfer with a content loss coefficient of 0 and a style loss coefficient of 1, we find that using 0.5 content and style loss coefficients decreases the mCE by 0.6%. Later, we show that SIN and AUGMIX can be combined. All models are trained from scratch, except MaxBlur Pooling models which has trained models available. Training Setup. Methods are trained with ResNet-50 and we follow the standard training scheme of , in which we linearly scale the learning rate with the batch size, and use a learning rate warm-up for the first 5 epochs, and AutoAugment and AUGMIX train for 180 epochs. All input images are first pre-processed with standard random cropping horizontal mirroring. Table 2: Clean Error, Corruption Error (CE), and mCE values for various methods on ImageNet-C. The mCE value is computed by averaging across all 15 CE values. AUGMIX reduces corruption error while improving clean accuracy, and it can be combined with SIN for greater corruption robustness. Results. Our method achieves 68.4% mCE as shown in Table 2, down from the baseline 80.6% mCE. Additionally, we note that AUGMIX allows straightforward stacking with other methods such as SIN to achieve an even lower corruption error of 64.1% mCE. Other techniques such as AutoAugment* require much tuning, while ours does not. Across increasing severities of corruptions, our method also produces much more calibrated predictions measured by both the Brier Score and RMS Calibration Error as shown in Figure 7. As shown in Table 3, AUGMIX also achieves a state-of-the art on ImageNet-P at with an mFR of 37.4%, down from 57.2%. We demonstrate that scaling up AUGMIX from CIFAR to ImageNet also leads to state-of-the-art in robustness and uncertainty estimation. We locate the utility of AUGMIX in three factors: training set diversity, our Jensen-Shannon divergence consistency loss, and mixing. Improving training set diversity via increased variety of augmentations can greatly improve robustness. For instance, augmenting each example with a Table 3: ImageNet-P . The mean flipping rate is the average of the flipping rates across all 10 perturbation types. AUGMIX improves perturbation stability by approximately 20%. Figure 7: Uncertainty on ImageNet-C. Observe that under severe data shifts, the RMS calibration error with ensembles and AUGMIX is remarkably steady. Even though classification error increases, calibration is roughly preserved. Severity zero denotes clean data. randomly sampled augmentation chain decreases the error rate of Wide ResNet on CIFAR-10-C from 26.9% to 17.0% Table 4. Adding in the Jensen-Shannon divergence consistency loss drops error rate further to 14.7%. Mixing random augmentations without the Jenson-Shannon divergence loss gives us an error rate of 13.1%. Finally, re-introducing the Jensen-Shannon divergence gives us AUGMIX with an error rate of 11.2%. Note that adding even more mixing is not necessarily beneficial. For instance, applying AUGMIX on top of Mixup increases the error rate to 13.3%, possibly due to an increased chance of manifold intrusion . Hence AUGMIX's careful combination of variety, consistency loss, and mixing explain its performance. CIFAR-10-C Error Rate CIFAR-100-C Error Rate Table 4: Ablating components of AUGMIX on CIFAR-10-C and CIFAR-100-C. Variety through randomness, the Jensen-Shannon divergence (JSD) loss, and augmentation mixing confer robustness. AUGMIX is a data processing technique which mixes randomly generated augmentations and uses a Jensen-Shannon loss to enforce consistency. Our simple-to-implement technique obtains state-of-the-art performance on CIFAR-10/100-C, ImageNet-C, CIFAR-10/100-P, and ImageNet-P. AUGMIX models achieve state-of-the-art calibration and can maintain calibration even as the distribution shifts. We hope that AUGMIX will enable more reliable models, a necessity for models deployed in safety-critical environments. In this section we demonstrate that AUGMIX's hyperparameters are not highly sensitive, so that AUGMIX performs reliably without careful tuning. For this set of experiments, the baseline AUGMIX model trains for 90 epochs, has a mixing coefficient of α = 0.5, has 3 examples per Jensen-Shannon Divergence (1 clean image, 2 augmented images), has a chain depth stochastically varying from 1 to 3, and has k = 3 augmentation chains. Figure 8 shows that the performance of various AUGMIX models with different hyperparameters. Under these hyperparameter changes, the mCE does not change substantially. A commonly mentioned hypothesis for the lack of robustness of deep neural networks is that they readily latch onto spurious high-frequency correlations that exist in the data. In order to better understand the reliance of models to such correlations, we measure model sensitivity to additive noise at differing frequencies. We create a 32 × 32 sensitivity heatmap. That is, we add a total of 32 × 32 Fourier basis vectors to the CIFAR-10 test set, one at a time, and record the ing error rate after adding each Fourier basis vector. Each point in the heatmap shows the error rate on the CIFAR-10 test set after it has been perturbed by a single Fourier basis vector. Points corresponding to low frequency vectors are shown in the center of the heatmap, whereas high frequency vectors are farther from the center. For further details on Fourier sensitivity analysis, we refer the reader to Section 2 of. In Figure 9 we observe that the baseline model is robust to low frequency perturbations but severely lacks robustness to high frequency perturbations, where error rates exceed 80%. The model trained with Cutout shows a similar lack of robustness. In contrast, the model trained with AUGMIX maintains robustness to low frequency perturbations, and on the mid and high frequencies AUGMIX is conspicuously more robust. The augmentation operations we use for AUGMIX are shown in Figure 10. We do not use augmentations such as contrast, color, brightness, sharpness, and Cutout as they may overlap with ImageNet-C test set corruptions. We should note that augmentation choice requires additional care. show that blithely applying augmentations can potentially cause augmented images to take different classes. Figure 11 shows how histogram color swapping augmentation may change a bird's class, leading to a manifold intrusion. Manifold Intrusion from Color Augmentation Figure 11: An illustration of manifold intrusion , where histogram color augmentation can change the image's class. We include various additional for CIFAR-10, CIFAR-10-C and CIFAR-10-P below. Figure 12 reports accuracy for each corruption, Table 5 reports calibration for various architectures and Table 6 reports clean error and mFR. We refer to Section 4.1 for details about the architecture and training setup. Table 6: CIFAR-10 Clean Error and CIFAR-10-P mean Flip Probability. All values are percentages. While adversarial training performs well on CIFAR-10-P, it induces a substantial drop in accuracy (increase in error) on clean CIFAR-10 where AUGMIX does not. Due to the finite size of empirical test sets, the RMS Calibration Error must be estimated by partitioning all n test set examples into b contiguous bins {B 1, B 2, . . ., B b} ordered by prediction confidence. In this work we use bins which contain 100 predictions, so that we adaptively partition confidence scores on the interval (; b). Other works partition the interval with 15 bins of uniform length . With these b bins, we estimate the RMS Calibration Error empirically with the formula This is separate from classification error because a random classifier with an approximately uniform posterior distribution is approximately calibrated. Also note that adding the "refinement" E C [(P(Y = Y |C = c)(1 − (P(Y =Ŷ |C = c))] to the square of the RMS Calibration Error gives us the Brier Score .

Represent the TLDR for finding the corresponding scientific article

We obtain state-of-the-art on robustness to data shifts, and we maintain calibration under data shift even though even when accuracy drops

represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a new, efficient algorithm to construct adversarial examples by means of deformations, rather than additive perturbations.


Question: Represent the article for finding a single-sentence TLDR.

Question: Automatic Piano Fingering is a hard task which computers can learn using data. As data collection is hard and expensive, we propose to automate this process by automatically extracting fingerings from public videos and MIDI files, using computer-vision techniques. Running this process on 90 videos in the largest dataset for piano fingering with more than 150K notes. We show that when running a previously proposed model for automatic piano fingering on our dataset and then fine-tuning it on manually labeled piano fingering data, we achieve state-of-the-art . In addition to the fingering extraction method, we also introduce a novel method for transferring deep-learning computer-vision models to work on out-of-domain data, by fine-tuning it on out-of-domain augmentation proposed by a Generative Adversarial Network (GAN). For demonstration, we anonymously release a visualization of the output of our process for a single video on https://youtu.be/Gfs1UWQhr5Q Learning to play the piano is a hard task taking years to master. One of the challenging aspects when learning a new piece is the fingering choice in which to play each note. While beginner booklets contain many fingering suggestions, advanced pieces often contain none or a select few. Automatic prediction of PIANO-FINGERING can be a useful addition to new piano learners, to ease the learning process of new pieces. As manually labeling fingering for different sheet music is an exhausting and expensive task 1, In practice previous work (; ; ; ;) used very few tagged pieces for evaluation, with minimal or no training data. In this paper, we propose an automatic, low-cost method for detecting PIANO-FINGERING from piano playing performances captured on videos which allows training modern -data-hungry -neural networks. We introduce a novel pipeline that adapts and combines several deep learning methods which lead to an automatic labeled PIANO-FINGERING dataset. Our method can serve two purposes: an automatic "transcript" method that detects PIANO-FINGERING from video and MIDI files, when these are available, and serve as a dataset for training models and then generalize to new pieces. Given a video and a MIDI file, our system produces a probability distribution over the fingers for each played. Running this process on large corpora of piano pieces played by different artists, yields a total of 90 automatically finger-tagged pieces (containing 155,107 notes in total) and in the first public large scale PIANO-FINGERING dataset, which we name APFD. This dataset will grow over time, as more videos are uploaded to YouTube. We provide empirical evidence that APFD is valuable, both by evaluating a model trained on it over manually labeled videos, as well as its usefulness by fine-tuning the model on a manually created dataset, which achieves state-of-the-art . The process of extracting PIANO-FINGERING from videos alone is a hard task as it needs to detect keyboard presses, which are often subtle even for the human eye. We, therefore, turn to MIDI files to obtain this information. The extraction steps are as follows: We begin by locating the keyboard and identify each key on the keyboard (§3.2). Then, we identify the playing hands on top of the keyboard (§3.3), and detect the fingers given the hands bounding boxes (§3.4). Next, we align between the MIDI file and its corresponding video (§3.6) and finally assign for every pressed note, the finger which was most likely used to play it (§3.5). Albeit the expectation from steps like hand detection and pose estimation, which were extensively studied in the computer-vision literature, we find that in practice, state-of-the-art models do not excel in these tasks for our scenario. We therefore address these weaknesses by fine-tuning an object detection model §3.3 on a new dataset we introduce and train a CycleGAN to address the different lighting scenarios with the pose estimation model §3.4. PIANO-FINGERING was previously studied in multiple disciplines, such as music theory and computer animation (; ; ; ; ;). The fingering prediction task is formalized as follows: Given a sequence of notes, associate each note with a finger from the set {1, 2, 3, 4, 5} × {L, R}. This is subject to constraints such as the positions of each hand, anatomical plausibility of transitioning between two fingers, the hands' size, etc. Each fingering sequence has a cost, which is derived from a transition cost between two fingers. Early work modeled fingering prediction as a search problem, where the objective is to find the optimal fingering sequence to play all the notes with. A naive approach to finding the best sequence is to exhaustively evaluate all possible transitions between one note to another which is not computationally feasible. By defining a transition matrix corresponding to the probability or "difficulty" of transitioning from one note to another -one can calculate a cost function, which defines the predicted sequence likelihood. Using a search algorithm on top of the transitions allows finding a globally optimal solution. This solution is not practical as well, due to the exponential complexity, and therefore heuristics or pruning are employed to reduce the space complexity. The transition matrix can be manually defined by heuristics or personal estimation , or instead, not relying on a pre-defined set of rules, and use a Hidden Markov Model (HMM) to learn the transitions . In practice, leaves the parameter learning to future work, and instead they manually fine-tune the transition matrix. On top of the transition matrix, practitioners suggested using dynamic programming algorithms to solve the search . Another option to solve the huge search space is to use a search algorithm such as Tabu search or variable neighborhood search (Mladenović &), to find a global plausible solution (; . These works are either limited by the defined transition rules, or by making different assumptions to facilitate the search space. Such assumptions come in the form of limiting the predictions to a single hand, limiting the evaluation pieces to contain no chords, rests or substantial lengths during which player can take their hand off the keyboard. Furthermore, all of these works have very small evaluation sets, which in practice makes it hard to compare different approaches, and does not allow to use more advanced models such as neural networks. In this work, we continue the transition of search-based methods that optimize a set of constraints with learning methods that try to imitate human behavior by the use of large datasets. In practice, these methods require lots of training data to achieve good performance, and fingering labeling is particularly expensive and hard to obtain. One way to automatically gather rich fingering data with full hand pose estimation is by using motion capture (MOCAP) gloves when playing the piano. suggests a rule-based and data-based hybrid method, initially estimating fingering decisions using a Directed Acyclic Graph (DAG) based on rule-based comfort constraints which are smoothed using data recorded from limited playing sessions with motion capture gloves. As MOCAP requires special equipment and may affect the comfort of the player, other work, tried to automatically detect piano fingering from video and MIDI files. The pianist's fingernails were laid with colorful markers, which were detected by a computer vision program. As some occlusions can occur, they used some rules to correct the detected fingering. In practice, they implemented the system with a camera capturing only 2 octaves (out of 8) and performed a very limited evaluation. The rules they used are simple (such as: restricting one finger per played note, two successive notes cannot be played with the same finger), but far from capturing real-world scenarios. Previous methods for automatically collecting data were costly, as apart of the equipment needed during the piece playing, and the data-collectors had to pay the participating pianists. In our work, we rely solely on videos from YouTube, meaning costs remain minimal with the ability to scale up to new videos. released a relatively large dataset of manually labeled PIANO-FINGERING by one to six annotators, consisting of 150 pieces, with partially annotated scores (324 notes per piece on average) with a total of 48,726 notes matched with 100,044 tags from multiple annotators. This is the largest annotated PIANO-FINGERING corpus to date and a valuable resource for this field. The authors propose multiple methods for modeling the task of PIANO-FINGERING, including HMMs and neural networks, and report the best performance with an HMM-based model. In this work, we use their dataset as a gold dataset for comparison and adapt their model to compare to our automatically generated dataset. There is a genre of online videos in which people upload piano performances where both the piano and the hands are visible. On some channels, people not only include the video but also the MIDI file recorded while playing the piece. We propose to use machine learning techniques to extract fingering information from such videos, enabling the creation of a large dataset of pieces and their fingering information. This requires the orchestration and adaptation of several techniques, which we describe below. The final output we produce is demonstrated in Figure 1, where we colored both the fingers and the played notes based on the pose-estimation model (§3.4) and the predicted fingers that played them (§3.5). Note that the ring fingers of both hands as well as the index finger of the left hand and the middle finger of the right hand do not press any note in this particular frame, but may play a note in others. We get the information of played notes from the MIDI events. We extract videos from youtube.com, played by different piano players on a specific channel containing both video and MIDI files. In these videos, the piano is filmed in a horizontal angle directly to the keyboard, from which both the keyboard and hands are displayed (as can be seen in Figure 1). MIDI files A standard protocol for the interchange of musical information between musical instruments, synthesizers, and computers Musical Instrument Digital Interface (MIDI) is a standard format for the interchange of musical information between electronic musical instruments. It consists of a sequence of events describing actions to carry out, when, and allows for additional attributes. In the setup of piano recording, it records what note was played in what time for how long and its pressure strength (velocity). We only use videos that come along with a MIDI file, and use it as the source for the played notes and their timestamp. To allow a correct fingering assignment, we first have to find the keyboard and the bounding boxes of the keys. We detect the keyboard as the largest continuous bright area in the video and identify key boundaries using standard image processing techniques, taking into account the expected number of keys and their predictable location and clear boundaries. For robustness and in order to handle the interfering hands that periodically hide parts of the piano, we combine information from multiple random frames by averaging the predictions from each frame. A straightforward approach for getting fingers locations in an image is to use a pose estimation model directly on the entire image. In practice, common methods for full-body pose estimation such as OpenPose containing hand pose estimation, make assumptions about the wrist and elbow locations to automatically approximate the hands' locations. In the case of piano playing, the elbow does not appear in the videos, therefore these systems don't work. We instead, turn to a pipeline approach where we start by detecting the hands, cropping them, and passing the cropped frames to a pose estimation model that expects the hand to be in the middle of the frame. Object Detection (; ; a; b), and specifically Hand Detection Kölsch &; ) are well studied subjects. However, out of the published work providing source code, the code was either complicated to run (e.g. versioning, mobile-only builds, supporting specific GPU, etc.), containing private datasets, or only detecting hands with no distinction between left and right, which is important in our case. We, therefore, created a small dataset with random frames from different videos, corresponding to 476 hands in total evenly split between left and right 2. We then fine-tuned a pre-trained object detection model (Inception v2 , based on Faster R-CNN , trained on COCO challenge ) on our new dataset. The fine-tuned model works reasonably well and some hand detection bounding boxes are presented in Figure 1. We release this new dataset and the trained model alongside the rest of the resources developed in this work. Having a working hand-detection model, we perform hand detection on every frame in the video. If more than two hands are detected, we take the 2 highest probability defections as the correct bounding boxes. If two hands are detected with the same label ("left-left" or "right-right"), we discard the model's label, and instead choose the leftmost bounding box to have the label "left" and the other to have the label "right" -which is the most common position of hands on the piano. Having the bounding box of each hand is not enough, as in order to assign fingers to notes we need the hand's pose. How can we detect fingers that pressed the different keys? We turn to pose estimation models, a well-studied subject in computer vision and use standard models . Using off-the-shelve pose estimation model, turned to often fail in our scenario. Some failure example are presented in Figure 2c where the first pose is estimated correctly, but the rest either have wrong finger positions, shorter, or broken fingers. The videos we use contain visual effects (e.g. LED lights are turned on every time a key is pressed), and such the pose-estimation models exhibit two failures: when the LED colors are warm (red, orange, yellow), the model sees the light as an extension to the finger, and as such poorly estimates where the keypoints are for each finger; when the LED colors are cool (green, blue, white), the over-saturation seems to overwhelm the model, and it mistakenly estimates the hand's pose as much shorter, considering the lit parts of the piano as part of the . Some examples of these errors are presented in Figure 2c. Furthermore, the videos are usually very dark, high-contrast, and blurry due to motion blur, which standard datasets (; ; and models trained on top of them rarely encounter. Given the observation that pose-estimation works well on well lid-images, how can we augment the pose estimation model to other lighting situations? This scenario is similar to sim2real works (; Gupta & Booher;) where one wants to transfer a model from simulations to the real-world. These works learn a mapping function G 1: T → S that transfer instances x i from the target domain T (the real-world) into the source domain S (the simulation), where the mapping is usually achieved by employing a CycleGan . Then, models which are trained on the source domain are used on the transformation of the target domain G 1 (x i) and manage to generalize on the target domain. In our setup, we seek a mapping G 2: S → T that transforms the source domain (i.e the well-lid videos) into the target data (i.e the challenging lighted scenarios). After obtaining the transformation function G 2, we employ the pose estimation model f on the source domain, use the transformation separately, and align the prediction to the new representation. This novel setup benefits from performance boost as we only use the transformation function offline, before training and avoid using it for every prediction. We also benefit of better generalization as we keep good performance on the source domain, and gain major performance on the target domain. We manually detect videos and assign them into their group (well-lit or poorly-lit). Then, we automatically detect and crop hands from random frames, ing in 21,747 well-lit hands, and 12,832 poorly lit hands. We then trained a CycleGAN for multiple epochs and chose 15 training checkpoints that produced different lighting scenarios (some examples can be seen in Figure 2). We then fine-tune a pose-estimation model on the original, well lit frames, and on the 15 transformed frames. This procedure in a model that is robust to different lighting scenarios, as we show in Figures 2b and 2d, demonstrating its performance on different lighting scenarios. Given that we know which notes were pressed in any given frame (see §3.6 below), there is still uncertainty as to which finger pressed them. This uncertainty either comes from imperfect pose estimation, or multiple fingers located on top of a single note. We model the finger press estimation by calculating the probability of a specific finger to have been used, given the hand's pose and the pressed note: where i ∈ for the 5 fingers, h j ∈ {h l, h r} stands for the hand being used (left or right) and n k ∈ corresponding to the played key. We chose to estimate the pressed fingers as a Gaussian distribution N (µ, σ 2), where µ and σ are defined as follows: µ is the center of the key on the x axis and σ is its width. The score of each finger given a note in a specific frame is defined as: The probability of a given finger to have played a note given a note and a frame: (normalizing g for all fingers) As most keyboard presses last more than one frame, we make use of multiple frames to overcome some of the errors from previous steps and to estimate a more accurate prediction. For this reason, we aggregate the frames that were used during a key press. We treat the first frame as the main signal point, and assign each successive frame an exponentially declining weight Σ n l=1 0.5 l Figure 3: Precision and recall based on confidence for all 6,894 manually tagged notes As finger changes can occur in later frames. Finally, we normalize the weighted sum of probabilities to achieve a probability distribution for all frames. In our dataset, we release all probabilities for each played note, along with the maximum likelihood finger estimation. We define the "confidence" score of the extraction from a single piece, as the product of the highest probability for a finger for each note. Figure 3 shows the precision and recall of the predictions based on a cutoff for the highest probability of the note. We see a positive correlation between confidence threshold and precision, and a negative correlation between confidence and recall, meaning we can get relatively high precision for a small number of notes, or relatively low precision for a high number of notes. We consider the MIDI and video files to be complementary, as they were recorded simultaneously. The MIDI files are the source to which keys were pressed, in what time, and for how long. The videos are the source for the piano, hands, and fingers locations. These two sources are not synchronized, but as they depict the same piano performance, a perfect alignment exist (up to the video frequency resolution). We extract the audio track from the video, and treat the first audio peak as the beginning of the piece, clipping the prior part of the video and aligning it with the first event from the MIDI file. In practice, we determine this starting point as the first point in the signal where the signal amplitude is higher than a fifth of the mean absolute value of the entire signal. This heuristic achieves a reasonable alignment, but we observe some alignment mismatch of 80-200ms. We tackle the misalignment by using a signal from the final system confidence (Section 3.5), where every piece gets a final score estimation after running the whole process, depicting the system confidence on the predicted notes. We look for an alignment that maximizes the system confidence over the entire piece: where M IDI t0 is the starting time of the MIDI file, and V ideo tj is the alignment time of the video. V ideo t0 is obtained by the heuristic alignment described in the previous paragraph. We use the confidence score as a proxy of the alignment precision and search the alignment that maximizes the confidence score of the system. More specifically, given the initial offset from the audio-MIDI alignment, we take a window of 1 second in frames (usually 25) for each side and compute the score of the final system on the entire piece. We choose the offset that in the best confidence score as the alignment offset. We follow the methods described in this section, and use it to label 90 piano pieces from 42 different composers with 155,107 notes in total. On average, each piece contains 1,723 notes. In this section, we present multiple evaluations of our overall system. We begin by evaluating the entire process where we assess how the overall system performs on predicting the pressed fingering. Next, we use the dataset we collected and train a PIANO-FINGERING model. We fine-tune this model on a previously manually-annotated dataset for PIANO-FINGERING and show using our data achieves better performance. As piano pieces are usually played by two hands, we avoid modeling each hand separately, and instead use their symmetry property, and simply flip one hand's notes, matching it to the piano scale, following previous work practice . For evaluation, we use the match rate between the prediction and the ground truth. For cases where there is a single ground truth, this is equivalent to accuracy measurement. When more than one labeling is available, we simply average the accuracies with each labeling. As the pose estimation is one of the major components directly affecting our system's performance, we isolate this part in order to estimate the gain from fine-tuning using the CycleGAN (Section 3.4). We manually annotated five random pieces from our dataset by marking the pressing finger for each played note in the video. Then, by using our system (§3.5), we estimate what finger was used for each key. We use the confidence score produced by the model as a threshold to use or discard the key prediction of the model, and report precision, recall and F1 scores of multiple thresholds. Moreover, we compare the scores of these pieces with and without using the CycleGAN. We do so for the five annotated pieces and report these in Table 1. When considering a high confidence score (>90%) both the pre-trained and fine-tuned models correctly mark all considered notes (which consist of between 34-36% of the data). However, when considering decreasing confidences, the fine-tuned model manages to achieve higher precision and higher recall, contributing to an overall higher f1 score. With no confidence threshold (i.e using all fingering predictions), the pre-trained model achieves 93% F1, while the fine-tuned one achieves 97% F1, a 57% error reduction. In order to assess the value of APFD, we seek to show its usefulness on the end task: Automatic Piano Fingering. To this end, we train a standard sequence tagging neural model using our dataset, evaluating on the subset of the videos we manually annotated. Then, we fine-tune this model on PIG , a manually labeled dataset, on which we achieve superior than simply training on that dataset alone. We model the PIANO-FINGERING as a sequence labeling task, where given a sequence of notes n 1, n 2,..., n n we need to predict a sequence of fingering: y 1, y 2,..., y n, where y i ∈ {1, 2, 3, 4, 5} corresponding to 5 fingers of one hand. We employ a standard sequence tagging technique, by embedding each note and using a BiLSTM on top of it. On every contextualized note we then use a Multi-Layer-Perceptron (MLP) to predict the label. The model is trained to minimize cross-entropy loss. This is the same model used in , referred to as DNN (LSTM). didn't use a development set, therefore in this work, we leave 1 piece from the training set and make it a development set. Our dataset-APFD-is composed of 90 pieces, which we split into 75/10 for training and development sets respectively, and use the 5 manually annotated pieces as a test set. We note that the development set is silver data (automatically annotated) and probably contains mistakes. The are summarized in Table 1. We run the same architecture by with some different hyperparameters and achieve 71.4%/64.1% on our and PIG's test set respectively. To evaluate the usefulness of our data to PIG's data, we use the model trained on our silver data and fine-tune it on PIG. This in 66.8% accuracy, 2.3% above the previous state-of-the-art model which was achieved by an HMM . We attribute this gain in performance to our dataset, which both increases the number of training examples and allows to train bigger neural models which excel with more training examples. We also experiment in the opposite direction and fine-tune the model trained on PIG with our data, which in 73.6% accuracy, which is better than training on our data alone, achieving 73.2% accuracy. In this work, we present an automatic method for detecting PIANO-FINGERING from MIDI and video files of a piano performance. We employ this method on a large set of videos, and create the first large scale PIANO-FINGERING dataset, containing 90 unique pieces, with 155,107 notes in total. We show this dataset-although being noisy-is valuable, by training a neural network model on it, fine-tuning on a gold dataset, where we achieve state-of-the-art . In future work, we intend to improve the data collection by improving the pose-estimation model, better handling high speed movements and the proximity of the hands, which often cause errors in estimating their pose. Furthermore, we intend to design improved neural models that can take previous fingering predictions into account, in order to have a better global fingering transition.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We automatically extract fingering information from videos of piano performances, to be used in automatic fingering prediction models.

Represent the short summary for finding the corresponding scientific article (long)

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Domain adaptation refers to the problem of leveraging labeled data in a source domain to learn an accurate model in a target domain where labels are scarce or unavailable. A recent approach for finding a common representation of the two domains is via domain adversarial training , which attempts to induce a feature extractor that matches the source and target feature distributions in some feature space. However, domain adversarial training faces two critical limitations: 1) if the feature extraction function has high-capacity, then feature distribution matching is a weak constraint, 2) in non-conservative domain adaptation (where no single classifier can perform well in both the source and target domains), training the model to do well on the source domain hurts performance on the target domain. In this paper, we address these issues through the lens of the cluster assumption, i.e., decision boundaries should not cross high-density data regions. We propose two novel and related models: 1) the Virtual Adversarial Domain Adaptation (VADA) model, which combines domain adversarial training with a penalty term that punishes the violation the cluster assumption; 2) the Decision-boundary Iterative Refinement Training with a Teacher (DIRT-T) model, which takes the VADA model as initialization and employs natural gradient steps to further minimize the cluster assumption violation. Extensive empirical demonstrate that the combination of these two models significantly improve the state-of-the-art performance on the digit, traffic sign, and Wi-Fi recognition domain adaptation benchmarks. The development of deep neural networks has enabled impressive performance in a wide variety of machine learning tasks. However, these advancements often rely on the existence of a large amount of labeled training data. In many cases, direct access to vast quantities of labeled data for the task of interest (the target domain) is either costly or otherwise absent, but labels are readily available for related training sets (the source domain). A notable example of this scenario occurs when the source domain consists of richly-annotated synthetic or semi-synthetic data, but the target domain consists of unannotated real-world data BID28 ). However, the source data distribution is often dissimilar to the target data distribution, and the ing significant covariate shift is detrimental to the performance of the source-trained model when applied to the target domain BID27.Solving the covariate shift problem of this nature is an instance of domain adaptation BID2. In this paper, we consider a challenging setting of domain adaptation where 1) we are provided with fully-labeled source samples and completely-unlabeled target samples, and 2) the existence of a classifier in the hypothesis space with low generalization error in both source and target domains is not guaranteed. Borrowing approximately the terminology from BID2, we refer to this setting as unsupervised, non-conservative domain adaptation. We note that this is in contrast to conservative domain adaptation, where we assume our hypothesis space contains a classifier that performs well in both the source and target domains. To tackle unsupervised domain adaptation, BID9 proposed to constrain the classifier to only rely on domain-invariant features. This is achieved by training the classifier to perform well on the source domain while minimizing the divergence between features extracted from the source versus target domains. To achieve divergence minimization, BID9 employ domain adversarial training. We highlight two issues with this approach: 1) when the feature function has high-capacity and the source-target supports are disjoint, the domain-invariance constraint is potentially very weak (see Section 3), and 2) good generalization on the source domain hurts target performance in the non-conservative setting. BID24 addressed these issues by replacing domain adversarial training with asymmetric tri-training (ATT), which relies on the assumption that target samples that are labeled by a sourcetrained classifier with high confidence are correctly labeled by the source classifier. In this paper, we consider an orthogonal assumption: the cluster assumption BID5, that the input distribution contains separated data clusters and that data samples in the same cluster share the same class label. This assumption introduces an additional bias where we seek decision boundaries that do not go through high-density regions. Based on this intuition, we propose two novel models: 1) the Virtual Adversarial Domain Adaptation (VADA) model which incorporates an additional virtual adversarial training BID20 and conditional entropy loss to push the decision boundaries away from the empirical data, and 2) the Decision-boundary Iterative Refinement Training with a Teacher (DIRT-T) model which uses natural gradients to further refine the output of the VADA model while focusing purely on the target domain. We demonstrate that 1. In conservative domain adaptation, where the classifier is trained to perform well on the source domain, VADA can be used to further constrain the hypothesis space by penalizing violations of the cluster assumption, thereby improving domain adversarial training.2. In non-conservative domain adaptation, where we account for the mismatch between the source and target optimal classifiers, DIRT-T allows us to transition from a joint (source and target) classifier (VADA) to a better target domain classifier. Interestingly, we demonstrate the advantage of natural gradients in DIRT-T refinement steps. We report for domain adaptation in digits classification (MNIST-M, MNIST, SYN DIGITS, SVHN), traffic sign classification (SYN SIGNS, GTSRB), general object classification (STL-10, CIFAR-10), and Wi-Fi activity recognition . We show that, in nearly all experiments, VADA improves upon previous methods and that DIRT-T improves upon VADA, setting new state-of-the-art performances across a wide range of domain adaptation benchmarks. In adapting MNIST → SVHN, a very challenging task, we out-perform ATT by over 20%. Given the extensive literature on domain adaptation, we highlight several works most relevant to our paper. BID27; BID17 proposed to correct for covariate shift by re-weighting the source samples such that the discrepancy between the target distribution and reweighted source distribution is minimized. Such a procedure is problematic, however, if the source and target distributions do not contain sufficient overlap. BID13 BID16; BID9 proposed to instead project both distributions into some feature space and encourage distribution matching in the feature space. BID9 in particular encouraged feature matching via domain adversarial training, which corresponds approximately to Jensen-Shannon divergence minimization BID11. To better perform nonconservative domain adaptation, BID24 proposed to modify tri-training for domain adaptation, leveraging the assumption that highly-confident predictions are correct predictions . Several of aforementioned methods are based on BID1's theoretical analysis of domain adaptation, which states the following, Theorem 1 BID1 ) Let H be the hypothesis space and let (X s, s) and (X t, t) be the two domains and their corresponding generalization error functions. Then for any h ∈ H, DISPLAYFORM0 where d H∆H denotes the H∆H-distance between the domains X s and X t, DISPLAYFORM1 Intuitively, d H∆H measures the extent to which small changes to the hypothesis in the source domain can lead to large changes in the target domain. It is evident that d H∆H relates intimately to the complexity of the hypothesis space and the divergence between the source and target domains. For infinite-capacity models and domains with disjoint supports, d H∆H is maximal. A critical component to our paper is the cluster assumption, which states that decision boundaries should not cross high-density regions BID5. This assumption has been extensively studied and leveraged for semi-supervised learning, leading to proposals such as conditional entropy minimization BID12 and pseudo-labeling BID15. More recently, the cluster assumption has led to many successful deep semi-supervised learning algorithms such as semi-supervised generative adversarial networks, virtual adversarial training BID20, and self/temporal-ensembling BID14 BID29. Given the success of the cluster assumption in semi-supervised learning, it is natural to consider its application to domain adaptation. Indeed, BID0 formalized the cluster assumption through the lens of probabilistic Lipschitzness and proposed a nearest-neighbors model for domain adaptation. Our work extends this line of research by showing that the cluster assumption can be applied to deep neural networks to solve complex, high-dimensional domain adaptation problems. Independently of our work, BID8 demonstrated the application of selfensembling to domain adaptation. However, our work additionally considers the application of the cluster assumption to non-conservative domain adaptation. Before describing our model, we first highlight that domain adversarial training may not be sufficient for domain adaptation if the feature extraction function has high-capacity. Consider a classifier h θ, parameterized by θ, that maps inputs to the (K − 1)-simplex (denote as C), where K is the number of classes. Suppose the classifier h = g • f can be decomposed as the composite of an embedding function f θ: X → Z and embedding classifier g θ: Z → C. For the source domain, let D s be the joint distribution over input x and one-hot label y and let X s be the marginal input distribution. DISPLAYFORM0 where the supremum ranges over discriminators D: Z →. Then L y is the cross-entropy objective and D is a domain discriminator. Domain adversarial training minimizes the objective DISPLAYFORM1 where λ d is a weighting factor. Minimization of L d encourages the learning of a feature extractor f for which the Jensen-Shannon divergence between f (X s) and f (X t) is small. 2 BID9 suggest that successful adaptation tends to occur when the source generalization error and feature divergence are both small. It is easy, however, to construct situations where this suggestion fails. In particular, if f has infinitecapacity and the source-target supports are disjoint, then f can employ arbitrary transformations to the target domain so as to match the source feature distribution (see Appendix E for formalization).We verify empirically that, for sufficiently deep layers, jointly achieving small source generalization error and feature divergence does not imply high accuracy on the target task TAB6. Given the limitations of domain adversarial training, we wish to identify additional constraints that one can place on the model to achieve better, more reliable domain adaptation. In this paper, we apply the cluster assumption to domain adaptation. The cluster assumption states that the input distribution X contains clusters and that points in the same cluster come from the same class. This assumption has been extensively studied and applied successfully to a wide range of classification tasks (see Section 2). If the cluster assumption holds, the optimal decision boundaries should occur far away from data-dense regions in the space of X BID5. Following BID12, we achieve this behavior via minimization of the conditional entropy with respect to the target distribution, Intuitively, minimizing the conditional entropy forces the classifier to be confident on the unlabeled target data, thus driving the classifier's decision boundaries away from the target data BID12. In practice, the conditional entropy must be empirically estimated using the available data. However, BID12 note that this approximation breaks down if the classifier h is not locally-Lipschitz. Without the locally-Lipschitz constraint, the classifier is allowed to abruptly change its prediction in the vicinity of the training data points, which 1) in a unreliable empirical estimate of conditional entropy and 2) allows placement of the classifier decision boundaries close to the training samples even when the empirical conditional entropy is minimized. To prevent this, we propose to explicitly incorporate the locally-Lipschitz constraint via virtual adversarial training BID20 and add to the objective function the additional term DISPLAYFORM0 which enforces classifier consistency within the norm-ball neighborhood of each sample x. Note that virtual adversarial training can be applied with respect to either the target or source distributions. We can combine the conditional entropy minimization objective and domain adversarial training to yield min. DISPLAYFORM1 a basic combination of domain adversarial training and semi-supervised training objectives. We refer to this as the Virtual Adversarial Domain Adaptation (VADA) model. Empirically, we observed that the hyperparameters (λ d, λ s, λ t) are easy to choose and work well across multiple tasks (Appendix B).H∆H-Distance Minimization. VADA aligns well with the theory of domain adaptation provided in Theorem 1. Let the loss, DISPLAYFORM2 denote the degree to which the target-side cluster assumption is violated. Modulating λ t enables VADA to trade-off between hypotheses with low target-side cluster assumption violation and hypotheses with low source-side generalization error. Setting λ t > 0 allows rejection of hypotheses with high target-side cluster assumption violation. By rejecting such hypotheses from the hypothesis space H, VADA reduces d H∆H and yields a tighter bound on the target generalization error. We verify empirically that VADA achieves significant improvements over existing models on multiple domain adaptation benchmarks (Table 1). VADA DIRT-T In non-conservative domain adaptation, we assume the following inequality, DISPLAYFORM0 where (s, t) are generalization error functions for the source and target domains. This means that, for a given hypothesis class H, the optimal classifier in the source domain does not coincide with the optimal classifier in the target domain. We assume that the optimality gap in Eq. FORMULA0 from violation of the cluster assumption. In other words, we suppose that any source-optimal classifier drawn from our hypothesis space necessarily violates the cluster assumption in the target domain. Insofar as VADA is trained on the source domain, we hypothesize that a better hypothesis is achievable by introducing a secondary training phase that solely minimizes the target-side cluster assumption violation. Under this assumption, the natural solution is to initialize with the VADA model and then further minimize the cluster assumption violation in the target domain. In particular, we first use VADA to learn an initial classifier h θ0. Next, we incrementally push the classifier's decision boundaries away from data-dense regions by minimizing the target-side cluster assumption violation loss L t in Eq.. We denote this procedure Decision-boundary Iterative Refinement Training (DIRT). Stochastic gradient descent minimizes the loss L t by selecting gradient steps ∆θ according to the following objective, min. DISPLAYFORM0 DISPLAYFORM1 which defines the neighborhood in the parameter space. This notion of neighborhood is sensitive to the parameterization of the model; depending on the parameterization, a seemingly small step ∆θ may in a vastly different classifier. This contradicts our intention of incrementally and locally pushing the decision boundaries to a local conditional entropy minimum, which requires that the decision boundaries of h θ+∆θ stay close to that of h θ. It is therefore important to define a neighborhood that is parameterization-invariant. Following BID21, we instead select ∆θ using the following objective, min. DISPLAYFORM2 Each optimization step now solves for a gradient step ∆θ that minimizes the conditional entropy, subject to the constraint that the Kullback-Leibler divergence between h θ (x) and h θ+∆θ (x) is small for x ∼ X t. The corresponding Lagrangian suggests that one can instead minimize a sequence of optimization problems min. DISPLAYFORM3 that approximates the application of a series of natural gradient steps. In practice, each of the optimization problems in Eq. FORMULA0 can be solved approximately via a finite number of stochastic gradient descent steps. We denote the number of steps taken to be the refinement interval B. Similar to BID29, we use the Adam Optimizer with Polyak averaging BID22. We interpret h θn−1 as a (sub-optimal) teacher for the student model h θn, which is trained to stay close to the teacher model while seeking to reduce the cluster assumption violation. As a , we denote this model as Decision-boundary Iterative Refinement Training with a Teacher (DIRT-T).Weakly-Supervised Learning. This sequence of optimization problems has a natural interpretation that exposes a connection to weakly-supervised learning. In each optimization problem, the teacher model h θn−1 pseudo-labels the target samples with noisy labels. Rather than naively training the student model h θn on the noisy labels, the additional training signal L t allows the student model to place its decision boundaries further from the data. If the clustering assumption holds and the initial noisy labels are sufficiently similar to the true labels, conditional entropy minimization can improve the placement of the decision boundaries BID23. Adaptation. An alternative interpretation is that DIRT-T is the recursive extension of VADA, where the act of pseudo-labeling of the target distribution constructs a new "source" domain (i.e. target distribution X t with pseudo-labels). The sequence of optimization problems can then be seen as a sequence of non-conservative domain adaptation problems in which DISPLAYFORM0 is the true conditional label distribution in the target domain. Since d H∆H is strictly zero in this sequence of optimization problems, domain adversarial training is no longer necessary. Furthermore, if L t minimization does improve the student classifier, then the gap in Eq. should get smaller each time the source domain is updated. In principle, our method can be applied to any domain adaptation tasks so long as one can define a reasonable notion of neighborhood for virtual adversarial training BID19. For comparison against BID24 and BID8, we focus on visual domain adaptation and evaluate on MNIST, MNIST-M, Street View House Numbers (SVHN), Synthetic Digits (SYN DIGITS), Synthetic Traffic Signs (SYN SIGNS), the German Traffic Signs Recognition Benchmark (GTSRB), CIFAR-10, and STL-10. For non-visual domain adaptation, we evaluate on Wi-Fi activity recognition. Architecture We use a small CNN for the digits, traffic sign, and Wi-Fi domain adaptation experiments, and a larger CNN for domain adaptation between CIFAR-10 and STL-10. Both architectures are available in Appendix A. For fair comparison, we additionally report the performance of source-only baseline models and demonstrate that the significant improvements are attributable to our proposed method. Replacing gradient reversal. In contrast to BID9, which proposed to implement domain adversarial training via gradient reversal, we follow BID11 and instead optimize via alternating updates to the discriminator and encoder (see Appendix C).Instance normalization. We explored the application of instance normalization as an image preprocessing step. This procedure makes the classifier invariant to channel-wide shifts and rescaling of pixel intensities. A discussion of instance normalization for domain adaptation is provided in Appendix D. We show in Figure 3 the effect of applying instance normalization to the input image. Figure 3: Effect of applying instance normalization to the input image. In clockwise direction: MNIST-M, GTSRB, SVHN, and CIFAR-10. In each quadrant, the top row is the original image, and the bottom row is the instance-normalized image. Hyperparameters. For each task, we tuned the four hyperparameters (λ d, λ s, λ t, β) by randomly selecting 1000 labeled target samples from the training set and using that as our validation set. We observed that extensive hyperparameter-tuning is not necessary to achieve state-of-the-art performance. In all experiments with instance-normalized inputs, we restrict our hyperparameter search for each task to λ d = {0, 10 −2}, λ s = {0, 1}, λ t = {10 −2, 10 −1}. We fixed β = 10 −2. Note that the decision to turn (λ d, λ s) on or off that can often be determined a priori. A complete list of the hyperparameters is provided in Appendix B. Table 1: Test set accuracy on visual domain adaptation benchmarks. In all settings, both VADA and DIRT-T achieve state-of-the-art performance in all settings. MNIST → MNIST-M. We first evaluate the adaptation from MNIST to MNIST-M. MNIST-M is constructed by blending MNIST digits with random color patches from the BSDS500 dataset. MNIST ↔ SVHN. The distribution shift is exacerbated when adapting between MNIST and SVHN. Whereas MNIST consists of black-and-white handwritten digits, SVHN consists of crops of colored, street house numbers. Because MNIST has a significantly lower intrinsic dimensionality that SVHN, the adaptation from MNIST → SVHN is especially challenging when the input is not pre-processed via instance normalization. When instance normalization is applied, we achieve a strong state-ofthe-art performance 76.5% and an equally impressive margin-of-improvement over source-only of 35.6%. Interestingly, by reducing the refinement interval B and taking noisier natural gradient steps, we were occasionally able to achieve accuracies as high as 87%. However, due to the high-variance associated with this, we omit reporting this configuration in Table 1.SYN DIGITS → SVHN. The adaptation from SYN DIGITS → SVHN reflect a common adaptation problem of transferring from synthetic images to real images. The SYN DIGITS dataset consist of 500000 images generated from Windows fonts by varying the text, positioning, orientation, , stroke color, and the amount of blur. SYN SIGNS → GTSRB. This setting provides an additional demonstration of adapting from synthetic images to real images. Unlike SYN DIGITS → SVHN, SYN SIGNS → GTSRB contains 43 classes instead of 10.STL ↔ CIFAR. Both STL-10 and CIFAR-10 are 10-class image datasets. These two datasets contain nine overlapping classes. Following the procedure in BID8, we removed the non-overlapping classes ("frog" and "monkey") and reduce to a 9-class classification problem. We achieve state-of-the-art performance in both adaptation directions. In STL → CIFAR, we achieve a 11.7% margin-of-improvement and a performance accuracy of 73.3%. Note that because STL-10 contains a very small training set, it is difficult to estimate the conditional entropy, thus making DIRT-T unreliable for CIFAR → STL. Wi-Fi Activity Recognition. To evaluate the performance of our models on a non-visual domain adaptation task, we applied VADA and DIRT-T to the Wi-Fi Activity Recognition Dataset . The Wi-Fi Activity Recognition Dataset is a classification task that takes the WiFi Channel State Information (CSI) data stream as input x to predict motion activity within an indoor area as output y. Domain adaptation is necessary when the training and testing data are collected from different rooms, which we denote as Rooms A and B. TAB2 shows that VADA significantly improves classification accuracy compared to Source-Only and DANN by 17.3% and 15% respectively. However, DIRT-T does not lead to further improvements on this dataset. We perform experiments in Appendix F which suggests that VADA already achieves strong clustering in the target domain for this dataset, and therefore DIRT-T is not expected to yield further performance improvement. Table 3: Additional comparison of the margin of improvement computed by taking the reported performance of each model and subtracting the reported source-only performance in the respective papers. W.I.N.I. indicates "with instance-normalized input."Overall. We achieve state-of-the-art across all tasks. For a fairer comparison against ATT and the Π-model, Table 3 provides the improvement margin over the respective source-only performance reported in each paper. In four of the tasks (MNIST → MNIST-M, SVHN → MNIST, MNIST → SVHN, STL → CIFAR), we achieve substantial margin of improvement compared to previous models. In the remaining three tasks, our improvement margin over the source-only model is competitive against previous models. Our closest competitor is the Π-model. However, unlike the Π-model, we do not perform data augmentation. It is worth noting that DIRT-T consistently improves upon VADA. Since DIRT-T operates by incrementally pushing the decision boundaries away from the target domain data, it relies heavily on the cluster assumption. DIRT-T's empirical success therefore demonstrates the effectiveness of leveraging the cluster assumption in unsupervised domain adaptation with deep neural networks.6.3 ANALYSIS OF VADA AND DIRT-T To study the relative contribution of the virtual adversarial training in the VADA and DIRT-T objectives (Eq. FORMULA6 and Eq. respectively), we perform an extensive ablation analysis in Table 4. The removal of the virtual adversarial training component is denoted by the "no-vat" subscript. Our show that VADA no-vat is sufficient for out-performing DANN in all but one task. The further ability for DIRT-T no-vat to improve upon VADA no-vat demonstrates the effectiveness of conditional entropy minimization. Ultimately, in six of the seven tasks, both virtual adversarial training and conditional entropy minimization are essential for achieving the best performance. The empirical importance of incorporating virtual adversarial training shows that the locally-Lipschitz constraint is beneficial for pushing the classifier decision boundaries away from data. Table 4: Test set accuracy in ablation experiments, starting from the DANN model. The "no-vat" subscript denote models where the virtual adversarial training component is removed. When considering Eq. FORMULA0, it is natural to ask whether defining the neighborhood with respect to the classifier is truly necessary. In FIG2, we demonstrate in SVHN → MNIST and STL → CIFAR that removal of the KL-term negatively impacts the model. Since the MNIST data manifold is low-dimensional and contains easily identifiable clusters, applying naive gradient descent (Eq. FORMULA0) can also boost the test accuracy during initial training. However, without the KL constraint, the classifier can sometimes deviate significantly from the neighborhood of the previous classifier, and the ing spikes in the KL-term correspond to sharp drops in target test accuracy. In STL → CIFAR, where the data manifold is much more complex and contains less obvious clusters, naive gradient descent causes immediate decline in the target test accuracy. We further analyze the behavior of VADA and DIRT-T by showing T-SNE embeddings of the last hidden layer of the model trained to adapt from MNIST → SVHN. In FIG3, source-only training shows strong clustering of the MNIST samples (blue) and performs poorly on SVHN (red). VADA offers significant improvement and exhibits signs of clustering on SVHN. DIRT-T begins with the VADA initialization and further enhances the clustering, ing in the best performance on MNIST → SVHN. In TAB6, we applied domain adversarial training to various layers of a Domain Adversarial Neural Network BID9 trained to adapt MNIST → SVHN. With the exception of layers L − 2 and L − 0, which experienced training instability, the general observation is that as the layer gets deeper, the additional capacity of the corresponding embedding function allows better matching of the source and target distributions without hurting source generalization accuracy. This demonstrates that the combination of low divergence and high source accuracy does not imply better adaptation to the target domain. Interestingly, when the classifier is regularized to be locally-Lipschitz via VADA, the combination of low divergence and high source accuracy appears to correlate more strongly with better adaptation. In this paper, we presented two novel models for domain adaptation inspired by the cluster assumption. Our first model, VADA, performs domain adversarial training with an added term that penalizes violations of the cluster assumption. Our second model, DIRT-T, is an extension of VADA that recursively refines the VADA classifier by untethering the model from the source training signal and applying approximate natural gradients to further minimize the cluster assumption violation. Our experiments demonstrate the effectiveness of the cluster assumption: VADA achieves strong performance across several domain adaptation benchmarks, and DIRT-T further improves VADA performance. Our proposed models open up several possibilities for future work. One possibility is to apply DIRT-T to weakly supervised learning; another is to improve the natural gradient approximation via K-FAC BID18 and PPO BID25. Given the strong performance of our models, we also recommend them for other downstream domain adaptation applications. DISPLAYFORM0 Gaussian noise, σ = 1 DISPLAYFORM1 Gaussian noise, σ = 1 DISPLAYFORM2 We observed that extensive hyperparameter-tuning is not necessary to achieve state-of-the-art performance. To demonstrate this, we restrict our hyperparameter search for each task to λ d = {0, 10 −2}, λ s = {0, 1}, λ t = {10 −2, 10 −1}, in all experiments with instance-normalized inputs. We fixed β = 10 −2. Note that the decision to turn (λ d, λ s) on or off that can often be determined a priori based on prior belief regarding the extent to covariate shift. In the absence of such prior belief, a reliable choice is (λ d = 10 −2, λ s = 1, λ t = 10 −2, β = 10 −2). When the target domain is MNIST/MNIST-M, the task is sufficiently simple that we only allocate B = 500 iterations to each optimization problem in Eq.. In all other cases, we set the refinement interval B = 5000. We apply Adam Optimizer (learning rate = 0.001, β 1 = 0.5, β 2 = 0.999) with Polyak averaging (more accurately, we apply an exponential moving average with momentum = 0.998 to the parameter trajectory). VADA was trained for 80000 iterations and DIRT-T takes VADA as initialization and was trained for {20000, 40000, 60000, 80000} iterations, with number of iterations chosen as hyperparameter. We note from BID11 that the gradient of ∇ θ ln(1 − D(f θ (x))) is tends to have smaller norm than −∇ θ ln D(f θ (x)) during initial training since the latter rescales the gradient by 1/D(f θ (x)). Following this observation, we replace the gradient reversal procedure with alternating minimization of DISPLAYFORM0 The choice of using gradient reversal versus alternating minimization reflects a difference in choice of approximating the mini-max using saturating versus non-saturating optimization BID7. In some of our initial experiments, we observed the replacement of gradient reversal with alternating minimization stabilizes domain adversarial training. However, we encourage practitioners to try either optimization strategy when applying VADA. Theorem 1 suggests that we should identify ways of constraining the hypothesis space without hurting the global optimal classifier for the joint task. We propose to further constrain our model by introducing instance normalization as an image pre-processing step for the input data. Instance normalization was proposed for style transfer BID30 and applies the operation DISPLAYFORM0 where x (i) ∈ R H×W ×C denotes the i th sample with (H, W, C) corresponding to the height, width, and channel dimensions, and where µ, σ: R H×W ×C → R C are functions that compute the mean and standard deviation across the spatial dimensions. A notable property of instance normalization is that it is invariant to channel-wide scaling and shifting of the input elements. Formally, consider scaling and shift variables γ, β ∈ R C. If γ 0 and σ(DISPLAYFORM1 For visual data the application of instance normalization to the input layer makes the classifier invariant to channel-wide shifts and scaling of the pixel intensities. For most visual tasks, sensitivity to channel-wide pixel intensity changes is not critical to the success of the classifier. As such, instance normalization of the input may help reduce d H∆H without hurting the globally optimal classifier. Interestingly, Figure 3 shows that input instance normalization is not equivalent to gray-scaling, since color is partially preserved. To test the effect of instance normalization, we report both with and without the use of instance-normalized inputs. We denote the source and target distributions respectively as p s (x, y) and p t (x, y). Let the source covariate distribution p s (x) define the random variable X s that have support supp(X s) = X s and let (X t, X t) be analogously defined for the target domain. Both X s and X t are subsets of R n. Let p s (y) and p t (y) define probabilities over the support Y = {1, . . ., K}. We consider any embedding function f: R n → R m, where R m is the embedding space, and any embedding classifier g: R m → C, where C is the (K − 1)-simplex. We denote a classifier h = g • f has the composite of an embedding function with an embedding classifier. For simplicity, we restrict our analysis to the simple case where K = 2, i.e. where Y = {0, 1}. Furthermore, we assume that for any δ ∈, there exists a subset Ω ⊆ R n where p s (x ∈ Ω) = δ. We impose a similar condition on p t (x).For a joint distribution p(x, y), we denote the generalization error of a classifier as DISPLAYFORM0 Note that for a given classifier h: R n →, the corresponding hard classifier is k(x) = 1{h(x) > 0.5}. We further define the set Ω ⊆ R n such that DISPLAYFORM1 In a slight abuse of notation, we define the generalization error (Ω) with respect to Ω as DISPLAYFORM2 An optimal Ω * p is a partitioning of DISPLAYFORM3 such that generalization error under the distribution p(x, y) is minimized. E.1 GOOD TARGET-DOMAIN ACCURACY IS NOT GUARANTEED Domain adversarial training seeks to find a single classifier h used for both the source p s and target p t distributions. To do so, domain adversarial training sets up the objective DISPLAYFORM4 DISPLAYFORM5 where F and G are the hypothesis spaces for the embedding function and embedding classifier. Intuitively, domain adversarial training operates under the hypothesis that good source generalization error in conjunction with source-target feature matching implies good target generalization error. We shall see, however, that if X s ∩ X t = ∅ and F is sufficiently complex, this implication does not necessarily hold. Let F contain all functions mapping R n → R m, i.e. F has infinite capacity. Suppose G contains the function g(z) = 1{z = 1 m} and X s ∩ X t = ∅. We consider the set DISPLAYFORM6 Such a set of classifiers satisfies the feature-matching constraint while achieving source generalization error no worse than the optimal source-domain hard classifier. It suffices to show that H * includes hypotheses that perform poorly in the target domain. We first show H * is not an empty set by constructing an element of this set. Choose a partitioning Ω where DISPLAYFORM7 Consider the embedding function DISPLAYFORM8 Let g(z) = 1{z = 1 m}. It follows that the composite classifier DISPLAYFORM9 Next, we show that a classifier h ∈ H * does not necessarily achieve good target generalization error. Consider the partitioningΩ which solves the following optimization problem DISPLAYFORM10 DISPLAYFORM11 Such a partitioningΩ is the worst-case partitioning subject to the probability mass constraint. It follows that worse case h ∈ H * has generalization error pt (h) = max h∈H * pt (h) ≥ pt (hΩ).To provide intuition that pt (h) is potentially very large, consider hypothetical source and target domains where X s ∩ X t = ∅ and p t (x ∈ Ω * pt) = p s (x ∈ Ω * ps) = 0.5. The worst-case partitioning subject to the probability mass constraint is simplyΩ = R n \ Ω Define the embedding functions DISPLAYFORM12 f (x) = 1 m if (x ∈ X s ∩ Ω s) ∨ (x ∈ X t ∩ (R n \ Ω t)) 0 m otherwise. Let g (z) = g(z) = 1{z = 1 m}. It follows that the composite classifiers h = g • f and h = g • f are elements ofH.From the definition of d H∆H, we see that DISPLAYFORM13 TheH∆H-divergence thus achieves the maximum value of 2. Our analysis assumes infinite capacity embedding functions and the ability to solve optimization problems exactly. The empirical success of domain adversarial training suggests that the use of finite-capacity convolutional neural networks combined with stochastic gradient-based optimization provides the necessary regularization for domain adversarial training to work. The theoretical characterization of domain adversarial training in the case finite-capacity convolutional neural networks and gradient-based learning remains a challenging but important open research problem. To evaluate the performance of our models on a non-visual domain adaptation task, we applied VADA and DIRT-T to the Wi-Fi Activity Recognition Dataset . The Wi-Fi Activity Recognition Dataset is a classification task that takes the Wi-Fi Channel State Information (CSI) data stream as input x to predict motion activity within an indoor area as output y. The dataset collected the CSI data stream samples associated with seven activities, denoted as "bed", "fall", "walk", "pick up", "run", "sit down", and "stand up".However, the joint distribution over the CSI data stream and motion activity changes depending on the room in which the data was collected. Since the data was collected for multiple rooms, we selected two rooms (denoted here as Room A and Room B) and constructed the unsupervised domain adaptation task by using Room A as the source domain and Room B as the target domain. We compare the performance of DANN, VADA, and DIRT-T on the Wi-Fi domain adaptation task in TAB2, using the hyperparameters (λ d = 0, λ s = 0, λ t = 10 −2, β = 10 −2). TAB2 shows that VADA significantly improves classification accuracy compared to Source-Only and DANN. However, DIRT-T does not lead to further improvements on this dataset. We believe this is attributable to VADA successfully pushing the decision boundary away from data-dense regions in the target domain. As a , further application of DIRT-T would not lead to better decision boundaries. To validate this hypothesis, we visualize the t-SNE embeddings for VADA and DIRT-T in FIG4 and show that VADA is already capable of yielding strong clustering in the target domain. To verify that the decision boundary indeed did not change significantly, we additionally provide the confusion matrix between the VADA and DIRT-T predictions in the target domain (Fig. 7).

SOTA on unsupervised domain adaptation by leveraging the cluster assumption.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Parametric adversarial divergences implicitly define more meaningful task losses for generative modeling, we make parallels with structured prediction to study the properties of these divergences and their ability to encode the task of interest.