Question: represent the scientific article to find a short tldr

Question: Transforming one probability distribution to another is a powerful tool in Bayesian inference and machine learning. Some prominent examples are constrained-to-unconstrained transformations of distributions for use in Hamiltonian Monte-Carlo and constructing flexible and learnable densities such as normalizing flows. We present Bijectors.jl, a software package for transforming distributions implemented in Julia, available at github.com/TuringLang/Bijectors.jl. The package provides a flexible and composable way of implementing transformations of distributions without being tied to a computational framework. We demonstrate the use of Bijectors.jl on improving variational inference by encoding known statistical dependencies into the variational posterior using normalizing flows, providing a general approach to relaxing the mean-field assumption usually made in variational inference. When working with probability distributions in Bayesian inference and probabilistic machine learning, transforming one probability distribution to another comes up quite often. For example, when applying Hamiltonian Monte Carlo on constrained distributions, the constrained density is usually transformed to an unconstrained density for which the sampling is performed . Another example is to construct highly flexible and learnable densities often referred to as normalizing flows (; ;); for a review see. When a distribution P is transformed into some other distribution Q using some measurable function b, we write Q = b * P and say Q is the push-forward of P. When b is a differentiable bijection with a differentiable inverse, i.e. a diffeomorphism or a bijector , the induced or pushed-forward distribution Qit is obtained by a simple application of change of variables. Specifically, given a distribution P on some Ω ⊆ R d with density p: Ω → [0, ∞), and a bijector b: Ω →Ω for someΩ ⊆ R d, the induced or pushed forward distribution Q = b * P has density q(y) = p b −1 (y) |det J b −1 (y)| or q b(x) = p(x) |det J b (x)| As mentioned, one application of this idea is learnable bijectors such as normalizing flows. One particular family of normalizing flow which has received a lot of attention is coupling flows (; ;). The idea is to use certain parts of the input vector x, say, x I 1 to construct parameters for a bijector f (the coupling law), which is then applied to a different part of the input vector, say, x I 2. In full generality, a coupling flow c I 1,I 2, the transformation in a coupling flow, is defined c I 1,I 2 (· ; f, θ): x I 2 → f x I 2; θ(x I 1) y I 2 → f −1 y I 2; θ(y I 1) where I 1, I 2 ⊂ I:= {1, . . ., d} are disjoint. As long as f ·; θ(x I 1): R I 2 → R I 2 is a bijector, c I 1,I 2 is invertible since y I 1 = x I 1. Note the parameter-map θ can be arbitrarily complex. Bijectors.jl is a framework for creating and using bijectors in the Julia programming language. The main idea is to treat standard constrained-to-unconstrained bijectors, e.g. log: R → (0, ∞), and more complex and possibly parameterized bijectors, e.g. coupling flows, as the same just as they are mathematically the same. This turns out to be quite a useful abstraction allowing seamless interaction between standard and learnable bijectors, making something like automatic differentiation variational inference (ADVI;) easy to implement (see Source Code 1). Table 1 shows supported mathematical operations. Only b(x) and b −1 (y) need to be manually implemented for a new bijector b. Another example is the introduction of neural autoregressive flows (NAFs;) where the inverse autoregressive flow (IAF;) is extended by replacing the affine coupling law used in IAF with a monotonic deep neural network. Despite the novel introduction of neural network, in Bijectors.jl the only difference between IAF and NAF is the choice of Bijector as the coupling law. A summarization of the related work and how it compares to Bijectors.jl can be seen in Table 2. More detailed comparisons can be found in Appendix A.1. 1. This refers to the torch.distributions submodule. After our submission, another transformer module based on PyTorch was released in Pyro : pyro.distributions.transforms. At the time of writing, we have not yet done a thorough comparison with this. At a first glance its features seem similar in natu re to tensorflow.probability which can be found in Table 2. 2. Bijectors.jl is agnostic to array-types used, therefore GPU functionality is provided basically for free by using the independent package CuArrays.jl to construct the arrays. forward(Q) , using neural networks from Flux.jl to define the coupling law in a coupling flow, and so on. For more examples, see the project website. Inversed{Bijectors. Logit{Float64},0}(Bijectors. Logit{Float64}(0.0, 1.0)) 12 13 julia> # Works like a standard`Distribution1 We demonstrate how to use Bijectors.jl by a possible approach to relaxing the meanfield assumption commonly made in variational inference through the use of normalizing flows. We consider a simple two-dimensional Gaussian with known covariance matrix with non-zero off-diagonal entries, i.e. different components are dependent, defined as follows In this case we can obtain an analytical expression for the posterior p(m | {x i} n i=1 ), and can indeed observe that the covariance matrix has non-zero off-diagonals. In this case, the mean-field assumption often made in variational inference is incorrect. Recall that in variational inference the objective is to maximize the evidence lower bound (ELBO) of the variational posterior q(m) and the true posterior For the ELBO of a transformed distribution, see Appendix B.2. Here we propose using a mean-field multivariate normal as a starting point, and then combine this with coupling flows to encode the structure of the model we are approximating into the variational posterior at a low computational cost. The idea is to encode an undirected edge between the random variables m 1 and m 2 by adding directed mappings in both directions; we do this by composing coupling flows c {2},{1} and c {1},{2}. For the coupling flows, we experimented with two different coupling laws f, affine and the recently introduced rational-quadratic splines . The parameter maps θ 1 and θ 2, respectively, were defined by a simple neural network in both cases. The ing density, letting Q µ,σ be the distribution of an isotropic multivariate Gaussian with mean µ and variance σ, is given by We then optimized the ELBO w.r.t. the parameters of the neural networks θ 1, θ 2, µ and σ to obtain our variational posteriors. The of the standard mean-field VI (MFVI) and this particular normalizing flow VI (NFVI) applied to the model in Equation can be seen in Figure 1. Here we observe that NFVI captures the correlation structure of the true posterior in Figure 1 (a) while MFVI, as expected, fails to do so. This is also reflected in the value of the ELBO for the two approaches (see Appendix B.1). This can potentially provide a flexible approach to taking advantage of structure in the joint distribution when performing variational inference without introducing a large number of parameters in addition to the mean-field parameters. See Appendix B.1 for specifics of the experiment. We presented Bijectors.jl, a framework for working with bijectors and thus transformations of distributions. We then demonstrated the flexibility of Bijectors.jl in an application of introducing correlation structure to the mean-field ADVI approach. We believe Bijectors.jl will be a useful tool for future research, especially in exploring normalizing flows and their place in variational inference. An interesting note about the NF variational posterior we constructed is that it only requires a constant number of extra parameters on top of what is required by mean-field normal VI. This approach can be applied in more general settings where one has access to the directed acyclic graph (DAG) of the generative model we want to perform inference. Then this approach will scale linearly with the number of unique edges between random variables. It is also possible in cases where we have an undirected graph representing a model by simply adding a coupling in both directions. This would be very useful for tackling issues faced when using mean-field VI and would be of interest to explore further. For related work we have mainly compared against Tensorflow's tensorflow probability, which is used by other known packages such pymc4, and PyTorch's torch.distributions, which is used by packages such as pyro. Other frameworks which make heavy use of such transformations using their own implementations are stan, pymc3, and so on. But in these frameworks the transformations are mainly used to transform distributions from constrained to unconstrained and vice versa with little or no integration between those transformation and the more complex ones, e.g. normalizing flows. pymc3 for example support normalizing flows, but treat them differently from the constrained-to-unconstrained transformations. This means that composition between standard and parameterized transformations is not supported. Of particular note is the bijectors framework in tensorflow probability introduced in . One could argue that this was indeed the first work to take such a drastic approach to the separation of the determinism and stochasticity, allowing them to implement a lot of standard distributions as a TransformedDistribution. This framework was also one of the main motivations that got the authors of Bijectors.jl interested in making a similar framework in Julia. With that being said, other than the name, we have not set out to replicate tensorflow probability and most of the direct parallels were observed after-the-fact, e.g. a transformed distribution is defined by the TransformedDistribution type in both frameworks. Instead we believe that Julia is a language well-suited for such a framework and therefore one can innovate on the side of implementation. For example in Julia we can make use of code-generation or meta-programming to do program transformations in different parts of the framework, e.g. the composition b • b −1 is transformed into the identity function at compile time. Similar to tensorflow probability we can use higher-order bijectors to construct new bijectors. Examples of such are Inverse, Compose, and Stacked. A significant difference is that in Bijectors.jl, the constructors are rarely called explicitly by the user but instead through a completely intuitive interface, e.g. inv (b) gives you the Inverse, b1 • b2 gives you the composition of b1 and b2, stack(b1, b2) gives you the two bijectors "stacked" together. Moreover, if b actually has a "named" inverse, e.g. b = Exp, then inv(b) will in Log rather than some thin wrapper Inversed(Exp). Irregardless of whether the bijector has a named inverse or not, the dual-nature is exploited in compositions so that b • inv(b) in Identity. For type-stable code, this is all done at compile-time. A particularly nice one is the Stacked(bijectors, ranges) which allows the user to specify which parts (or ranges) of the input vector should be passed to which of the "stacked". For all methods acting on a Stacked the loop for iterating through the different ranges and applying the corresponding Bijector will be unrolled, meaning that this abstraction has a zero-cost overhead and the only cost is the evaluation of corresponding methods on for the bijectors it wraps. In a limited sense Bijectors.jl can do what is known as program transformations. A good example is b • b −1 ing in identity at compile-time for simple transformations which we have mentioned before. In tensorflow probability indeed b • b −1 is reduced to the identity mapping, not by collapsing the computational graph but instead by the use of caching. This means that when (b • b −1)(x) is evaluated, work will only be done for the b −1 (x) evaluation. When b −1 (x) is evaluated by b, the cached value x used to evaluate b −1 just before will be returned immediately. torch.distributions take a similar approach but because caching can come with its own issues, especially when used in conjunction with automatic differentiation, there are cases where it will fail, e.g. dependency reversal. In Bijectors.jl there are two parts of this story. First off, b • b −1 will, as noted earlier, be compiled to the identity map upon compilation, i.e. there is zero-overhead at run-time to this evaluation. But one nice property of the Tensorflow and PyTorch approach which uses caching is that one can write code that looks like In Bijectors.jl this has to be done manually by the user through the forward method for a TransformedDistribution. Recall from Table 1 that forward returns a 4-tuple (x, b(x), logabsdetjac (b, x), logpdf(q, b(x))) using the most efficient computation path. Therefore to replicate the above example in Bijectors.jl, we can do # Samples x from base and returns y = b(x) x, y, _, _ = forward(transformed_distribution) # do some more computation potentially involving y #... Therefore "caching" in Bijectors.jl cannot be done across function barriers at the time of writing (unless the function explicitly returns all values used). On the bright side one can explicitly do caching, making it more difficult to do something wrong in addition to the fact that the computation is transparent from the users perspective. Appendix B. Adding structure to mean-field VI using coupling flows In the experimental setup we generate data by fixing m = 0 and generating n = 100 samples from Equation. This ed in a posterior multivariate normal with covariance matrix This was done by design, as we specifically chose L = 10 0 10 10 to get a posterior covariance matrix with non-zero off-diagonals. Let q µ,σ denote the density of a multivariate Gaussian with mean µ and diagonal covariance σ, and b denote the coupling flow in Equation with f as a rational-quadratic spline (RQS) with K = 3 knot points and bin [−50, 50], θ as a neural network consisting of one layer with a (3K − 1) × 1 weight matrix and bias with identity activation, i.e. a simple affine transformation. See for more information on RQS. We use Distributions.jl for implementation of the Gaussian multivariate distribution . We then performed variational inference on the model in Equation ing in Figure 5 (c). The ing densities can be observed in Figure 3. Note that here we have used a slight abuse of notation writing max θ to mean "maximize wrt. parameters of θ". The expressions for the KL-divergence and the ELBO, which under the transformation by a bijector picks up an additional term, see Equation and Equation, respectively. In all cases we set the number of samples used in the Monte-Carlo estimate of the objective to be m = 50. In all cases we used a DecayedADAGrad from Turing.jl to perform gradient updates. This is a classical ADAGrad but with a decay for the accumulated gradient norms. This is to circumvent the possibility of large initial gradient norms bringing all subsequent optimization steps to practically zero step-size. For DecayedADAGrad we used a base step-size η = 0.001, post-factor decay β post = 1.0 and pre-factor decay β pre = 0.9, and we performed 5 000 optimization steps before terminating. In general we of course do not have access to the true posterior and so we cannot minimize the KL-divergence between the variational posterior and the true posterior directly, but instead have to do so implicitly by minimizing the ELBO. In theory there is no difference, but in practice one usually observe a significantly lower variance in the gradient estimates of the KL-divergence compared to the ELBO. We therefore also performed VI using the KLdivergence to verify that the NF did not lack the expressibility to capture the true posterior, but that the slight inaccuracy in the variational posterior obtained by maximizing the ELBO was indeed due to the variance in the gradient estimate. And, as expected, minimizing the KL-divergence directly in the MF-case did not provide much of a gain compared to maximizing the ELBO. Numerical for multiple runs where the ELBO was used as an objective can be seen in Table 3 and Figure 2; the NFVI approach consistently obtains lower KL divergence and a greater ELBO. The main quantity of interest is the KL-divergence which quantifies the difference between the variational posterior and the true posterior. The ELBO is a lower bound on the evidence and thus the actual values can vary widely across experiments. Additionally, the difference between the ELBO of two distributions with respect to the same set of observations is equal to the difference between KL-divergence on that set of observations, and so we gain no additional information of the difference between the variational posterior and the true posterior by looking at the ELBO. Therefore we visualize the KL-divergence instead of the ELBO in Figure 2, but still provide the numerical values for both in Table 3 Table 3: (Rational-Quadratic Spline coupling law) Exponentially smoothed estimates of the last 4000 (out of 5000) optimization steps. As can be seen in Figure 2, after the 1000th step is basically when the optimas are reached. Here the ELBO has been used as the objective. We also performed the same experiment using an affine transformation f as a coupling law. The setup is identical, but now θ is a neural network consisting of two layers; the first layer is a dense layer with 2 × 1 weight matrix and bias and ReLU activation, and the second layer is a dense layer with 2 × 2 weight matrix and bias and identity activation. In Flux.jl, which we have used for the neural network part of the bijector, this is given by Chain(Dense(1, 2, relu), Dense)) . As one can see in Table 4 and Figure 4, even with an affine coupling law we obtain very good approximations. Table 4: (Affine coupling law) Exponentially smoothed estimates of the last 4000 (out of 5000) optimization steps. As can be seen in Figure 2, after the 1000th step is basically when the optimas are reached. Here the ELBO has been used as the objective. Recall the definition of the Kullback-Leibler (KL) divergence, here relating the variational density q(z) and posterior p(z | {x} n i=1 ), As per usual, we can rewrite this where in the second-to-last equality we used the assumption that the observations are i.i.d. and in the last equality we used the fact that log p(x i) is independent of z for all i = 1,..., n. We can then arrange this into Observe that given a set of observations, the left-hand side is constant. Therefore we can minimize the KL-divergence by maximizing the remaining terms on the right-hand side of the equation, which we call the evidence lower bound (ELBO) Now suppose that the variational posterior q(z) is in fact a transformed distribution, say, with base density q 0 and using transformation b, i.e. Substituting these terms into the ELBO from Equation, we get This expression ise very useful when q 0 is a density which it is computationally cheap to sample from and we have an analytical expression for the entropy of q 0, e.g. if q 0 is the density of a mulitvariate Gaussian both of these conditions are satisfied. In practice we use a Monte-Carlo estimate of the ELBO where η k ∼ q 0 (η) for k = 1,..., m. From this we can then obtain a Monte-Carlo estimate of the gradient wrt. parameters.

represent the tldr for finding the corresponding scientific article

For instance, <<We present an end-to-end design methodology for efficient deep learning deployment. >>  to "We present an end-to-end design methodology for efficient deep learning deployment. Unlike previous methods that separately optimize the neural network architecture, pruning policy, and quantization policy, we jointly optimize them in an end-to-end manner. To deal with the larger design space it brings, we train a quantization-aware accuracy predictor that fed to the evolutionary search to select the best fit. We first generate a large dataset of <NN architecture, ImageNet accuracy> pairs without training each architecture, but by sampling a unified supernet. Then we use these data to train an accuracy predictor without quantization, further using predictor-transfer technique to get the quantization-aware predictor, which reduces the amount of post-quantization fine-tuning time. Extensive experiments on ImageNet show the benefits of the end-to-end methodology: it maintains the same accuracy (75.1%) as ResNet34 float model while saving 2.2× BitOps comparing with the 8-bit model; we obtain the same level accuracy as MobileNetV2+HAQ while achieving 2×/1.3× latency/energy saving; the end-to-end optimization outperforms separate optimizations using ProxylessNAS+AMC+HAQ by 2.3% accuracy while reducing orders of magnitude GPU hours and CO2 emission. Deep learning has prevailed in many real-world applications like autonomous driving, robotics, and mobile VR/AR, while efficiency is the key to bridge research and deployment. Given a constrained resource budget on the target hardware (e.g., latency, model size, and energy consumption), it requires an elaborated design of network architecture to achieve the optimal performance within the constraint. Traditionally, the deployment of efficient deep learning can be split into model architecture design and model compression (pruning and quantization). Some existing works (b; have shown that such a sequential pipeline can significantly reduce the cost of existing models. Nevertheless, careful hyper-parameter tuning is required to obtain optimal performance . The number of hyper-parameters grows exponentially when we consider the three stages in the pipeline together, which will soon exceed acceptable human labor bandwidth. To tackle the problem, recent works have applied AutoML techniques to automate the process. Researchers proposed Neural Architecture Search (NAS) (; ; a; b; ; ; a; b; ;) to automate the model design, outperforming the human-designed models by a large margin. Based on a similar technique, researchers adopt reinforcement learning to compress the model by automated pruning and automated quantization. However, optimizing these three factors in separate stages will lead to sub-optimal : e.g., the best network architecture for the full-precision model is not necessarily the optimal one after pruning and quantization. Besides, this three-step strategy also requires considerable search time and energy consumption . Therefore, we need a joint, end-to-end solution to optimize the deep learning model for a certain hardware platform. However, directly extending existing AutoML techniques to our end-to-end model optimization setting can be problematic. Firstly, the joint search space is cubic compared to stage-wise search, making the search difficult. Introducing pruning and quantization into the pipeline will also greatly increase the total search time, as both of them require time-consuming post-processing (e.g., finetuning) to get accuracy approximation ). Moreover, the search space of each step in pipeline is hard to be attested to be disentangle, and each step has its own optimization objective (eg. acc, latency, energy), so that the final policy of the pipeline always turns out to be sub-optimal. To this end, we proposed EMS, an end-to-end design method to solve this problem. Our approach is derived from one-shot NAS (; ; ; ; a;). We reorganize the traditional pipeline of "model design→pruning→quantization" into "architecture search + mixed-precision search". The former consists of both coarse-grained architecture search (topology, operator choice, etc.) and fine-grained channel search (replacing the traditional channel pruning ). The latter aims to find the optimal mixed-precision quantization policy trading off between accuracy and resource consumption. We work on both aspects to address the search efficiency. For architecture search, we proposed to train a highly flexible super network that supports not only the operator change but also fine-grained channel change, so that we can perform joint search over architecture and channel number. For the mixed-precision search, since quantized accuracy evaluation requires time-consuming fine-tuning, we instead use a predictor to predict the accuracy after quantization. Nevertheless, collecting data pairs for predictor training could be expensive (also requires fine-tuning). We proposed PredictorTransfer Technique to dramatically improve the sample efficiency. Our quantization-aware accuracy predictor is transferred from full-precision accuracy predictor, which is firstly trained on cheap data points collected using our flexible super network (evaluation only, no training required). Once the predictor P (arch, prune, quantization) is trained, we can perform search at ultra fast speed just using the predictor. With the above design, we are able to efficiently perform joint search over model architecture, channel number, and mixed-precision quantization. The predictor can also be used for new hardware and deployment scenarios, without training the whole system again. Extensive experiment shows the superiority of our method: while maintaining the same level of accuracy (75.1%) with ResNet34 float model, we achieve 2.2× reduction in BitOps compared to the 8-bit version; we obtain the same level accuracy as MobileNetV2+HAQ, and achieve 2×/1.3× latency/energy saving; our models outperform separate optimizations using ProxylessNAS+AMC+HAQ by 2.3% accuracy under same latency constraints, while reducing orders of magnitude GPU hours and CO 2 emission. The contributions of this paper are: • We devise an end-to-end methodology EMS to jointly perform NAS-pruning-quantization, thus unifying the conventionally separated stages into an integrated solution. • We propose a predictor-transfer method to tackle the high cost of the quantization-aware accuracy predictor's dataset collection NN architecture, quantization policy, accuracy. • Such end-to-end method can efficiently search efficient models. With the supernet and the quantization-aware accuracy predictor, it only takes minutes to search a compact model for a new platform, enabling automatic model adjustment in diverse deployment scenarios. Researchers have proposed various methods to accelerate the model inference, including architecture design (;, network pruning and network quantization (b). Neural Architecture Search. Tracing back to the development of NAS, one can see the reduction in the search time. ) use an RL agent to determine the cell-wise architecture. To efficiently search for the architecture, many later works viewed architecture searching as a path finding problem (a; b), it cuts down the search time by jointly training rather than iteratively training from scratch. Inspired by the path structure, some one-shot methods have been proposed to further leverage the network's weights in training time and begin to handle mixed-precision case for efficient deployment. Another line of works tries to grasp the information by a performance predictor , which reduces the frequent evaluation for target dataset when searching for the optimal. (b), SPOS: Single Path One-Shot , ChamNet , AMC , HAQ and EMS (Ours). EMS distinguishes from other works by directly searching mixed-precision architecture without extra interaction with target dataset. Pruning. Extensive works show the progresses achieved in pruning: in early time, researchers proposed fine-grained pruning (; 2016b) by cutting off the connections (i.e., elements) within the weight matrix. However, such kind of method is not friendly to the CPU and GPU and requires dedicated hardware (that supports sparse matrix multiplication) to perform the inference. Later, some researchers proposed channel-level pruning (; ; ; ; ; ;) by pruning the entire convolution channel based on some importance score (e.g., L1-norm) to enable acceleration on general-purpose hardware. However, both fine-grained pruning and channel-level pruning introduces an enormous search space as different layer has different sensitivities (e.g., the first convolution layer is very sensitive to be pruned as it extracts important low-level features; while the last layer can be easily pruned as it's very redundant). To this end, recent researches leverage the AutoML techniques to automate this exploration process and surpass the human design. Quantization. Quantization is a necessary technique to deploy the models on hardware platforms like FPGAs and mobile phones. (a) quantized the network weights to reduce the model size by grouping the weights using k-means. binarized the network weights into {−1, +1}; quantized the network using one bit for weights and two bits for activation; binarized each convolution filter into {−w, +w}; mapped the network weights into {−w N, 0, +w P} using two bits with a trainable range; explicitly regularized the loss perturbation and weight approximation error in a incremental way to quantize the network using binary or ternary weights. used 8-bit integers for both weights and activation for deployment on mobile devices. Some existing works explored the relationship between quantization and network architecture. HAQ proposed to leverage AutoML to determine the bit-width for a mixed-precision quantized model. A better trade-off can be achieved when different layers are quantized with different bits, showing the strong correlation between network architecture and quantization. Multi-Stage Optimization. Above methods are orthogonal to each other and a straightforward combination approach is to apply them sequentially in multiple stages i.e. NAS+Pruning+Quantization: • In the first stage, we can search the neural network architecture with the best accuracy on the target dataset (; b; a): • In the second stage, we can prune the channels in the model automatically : • In the third stage, we can quantize the model to mixed-precision to make full use of the emerging hardware architecture: However, this separation usually leads to a sub-optimal solution: e.g., the best neural architecture for the floating-point model may not be optimal for the quantized model. Moreover, frequent evaluations on the target dataset make such kind of methods time-costly: e.g., a typical pipeline as above can take about 300 GPU hours, making it hard for researchers with limited computation resources to do automatic design. Figure 1: An overview of our end-to-end design methodology. We first train an accuracy predictor for the full precision NN, then incrementally train an accuracy predictor for the quantized NN (predictor-transfer). Finally, evolutionary search is performed to find the specialized NN architecture that fits hardware constraints. Joint Optimization. Instead of optimizing NAS, pruning and quantization independently, joint optimization aims to find a balance among these configurations and search for the optimal strategy. To this end, the joint optimization objective can be formalized into: However, the search space of this new objective is tripled as original one, so it becomes challenging to perform joint optimization. We endeavor to unify NAS, pruning and quantization as joint optimization. The outline is: 1. Train a super network that covers a large search space and every sub-network can be directly extracted without re-training. 2. Build a quantization-aware accuracy predictor to predict quantized accuracy given a sub-network and quantization policy. 3. Construct a latency/energy lookup table and do resource constrained evolution search. Thereby, this joint optimization problem can be tackled in an end-to-end manner. Comparison with Recent Methods. The search space is quadratic when comparing to (b), since we need to take care of both architecture configuration and quantization policy rather than quantization policy only. Unlike , whose predictor only use full precision (FP) data to train, we face an unbalance ratio in full precision (FP) and mixed-precision (MP) data. Also, architecture configuration and quantization policy are orthogonal to each other, simply treating this problem as before will lead to a significant performance drop when training the predictor. Different from , which uses ResNet as super network's backbone and cannot handle a more efficient scenario when changing backbone into MobileNet due to large accuracy drop after quantization, our super network provides a more stable accuracy statistic after sampling network to do quantization. This gives us the opportunity to acquire quantization data simply by extract a sub-network and do quantization. The overall framework of our end-to-end design framework is shown in Figure 1. It consists of a highly flexible super network with fine-grained channels, an accuracy predictor, and evolution search jointly optimizing architecture, pruning, and quantization. Neural architecture search aims to find a good sub-network from a large search space. Traditionally, each sampled network is trained to obtain the actual accuracy , which is time consuming. Recent one-shot based NAS first trains a large, multi-branch network. At each time, a sub-network is extracted from the large network to directly evaluate the approximated accuracy. Such a large network is called super network. Since the choice of different layers in a deep neural network is largely independent, a popular way is to design multiple choices (e.g., kernel size, expansion ratios) for each layer. In this paper, we used a super network that supports different kernel sizes (i.e. 3, 5, 7) and channel number (i.e. 4×B to 6×B, 8 as internal, B is the base channel number in that block) in block level, and different depths (i.e. 2, 3, 4) in stage level. The combined search space contains more than 10 35 sub-networks, which is large enough to conduct neural architecture search. Properties of the Super Network. We also followed the one-shot setting to first build a super network and then perform search on top of it. To ensure efficient architecture search, we find that the super network needs to satisfy the following properties: For every extracted sub-network, the performance could be directly evaluated without re-training, so that the cost of training only need to be paid once. Support an extremely large and fine-grained search space to support channel number search. As we hope to incorporate pruning policy into architecture space, the super network not only needs to support different operators, but also fine-grained channel numbers (8 as interval). Thereby, the new space is significantly enlarged (nearly quadratic from 10 19 to 10 35). However, it is hard to achieve the two goals at the same time due to the nature of super network training: it is generally believed that if the search space gets too large (e.g., supporting fine-grained channel numbers), the accuracy approximation would be inaccurate (b). A large search space will in high variance when training the super network. To address the issue, We adopt progressive shrinking (PS) algorithm (a) to train the super network. Specifically, we first train a full sub-network with largest kernel sizes, channel numbers and depths in the super network, and use it as a teacher to progressively distill the smaller sub-networks sampled from the super network. During distillation, the trained sub-networks still update the weights to prevent accuracy loss. The PS algorithm effectively reduce the variance during super network training. By doing so, we can assure that the extracted sub-network from the super network preserves competitive accuracy without re-training. To reduce the cost for designs in various deployment scenarios, we propose to build a quantizationaware accuracy predictor P, which predicts the accuracy of the mixed-precision (MP) model based on architecture configurations and quantization policies. During search, we used the predicted accuracy acc = P (arch, prune, quantize) instead of the measured accuracy. The input to the predictor P is the encoding of the network architecture, the pruning strategy, and the quantization policy. Architecture and quantization policy encoding. We encode the network architecture block by block: for each building block (i.e. bottleneck residual block like MobileNetV2 We further concatenate the features of all blocks as the encoding of the whole network. Then for a 5-layer network, we can use a 75-dim(5×(3+4+2×4)=75) vector to represent such an encoding. In our setting, the choices of kernel sizes are, the choices of channel number depend on the base channel number for each block, and bitwidth choices are, there are 21 blocks in total to design. Accuracy Predictor. The predictor we use is a 3-layer feed-forward neural network with each embedding dim equaling to 400. As shown in the left of Figure 2, the input of the predictor is the one-hot encoding described above and the output is the predicted accuracy. Different from existing methods (a; b; a), our predictor based method does not require frequent evaluation of architecture on target dataset in the search phase. Once we have the predictor, we can integrate it with any search method (e.g. reinforcement learning, evolution, bayesian optimization, etc.) to perform end-to-end design over architecture-pruning-quantization at a negligible cost. However, the biggest challenge is how to collect a [architecture, quantization policy, accuracy] dataset to train the predictor for quantized models due to: 1) collecting quantized model's accuracy is time-consuming: fine-tuning is required to recover the accuracy after quantization, which takes about 0.2 GPU hours per data point. In fact, we find that 80k data pairs is a suitable size to train a good full precision accuracy predictor. If we collect a quantized dataset with the same size as the full-precision one, it can cost 16,000 GPU hours, which is far beyond affordable. 2) The quantization-aware accuracy predictor is harder to train than a traditional accuracy predictor on full-precision models: the architecture design and quantization policy affect network performance from two separate aspects, making it hard to model the mutual influence. Thus using traditional way to train quantization-aware accuracy predictor can in a significant performance drop (Table 2). Figure 2: Predictor-transfer technique. We start from a pre-trained full-precision predictor and add another input head (green square at bottom right) denoting quantization policy. Then fine-tune the quantization-aware accuracy predictor. Transfer Predictor to Quantized Models. Collecting a quantized NN dataset for training the predictor is difficult (needs finetuning), but collecting a full-precision NN dataset is easy: we can directly pick sub-networks from the super net and measure its accuracy. We propose the predictor-transfer technique to increase the sample efficiency and make up for the lack of data. As the order of accuracy before and after quantization is usually preserved, we first pre-train the predictor on a large-scale dataset to predict the accuracy of full-precision models, then transfer to quantized models. The quantized accuracy dataset is much smaller and we only perform short-term fine-tuning. As shown in Figure 2, we add the quantization bits (weights& activation) of the current block into the input embedding to build the quantization-aware accuracy predictor. We then further fine-tune the quantization-aware accuracy predictor using pre-trained FP predictor's weights as initialization. Since most of the weights are inherited from the full-precision predictor, the training requires much less data compared to training from scratch. As different hardware might have drastically different properties (e.g., cache size, level of parallelism), the optimal network architecture and quantization policy for one hardware is not necessarily the best for the other. Therefore, instead of relying on some indirect signals (e.g., BitOps), our optimization is directly based on the measured latency and energy on the target hardware. Measuring Latency and Energy. Evaluating each candidate policy on actual hardware can be very costly. Thanks to the sequential structure of neural network, we can approximate the latency (or energy) of the model by summing up the latency (or energy) of each layer. We can first build a lookup table containing the latency and energy of each layer under different architecture configurations and bit-widths. Afterwards, for any candidate policy, we can break it down and query the lookup table to directly calculate the latency (or energy) at negligible cost. In practice, we find that such practice can precisely approximate the actual inference cost. Resource-Constrained Evolution Search. We adopt the evolution-based architecture search to explore the best resource-constrained model. Based on this, we further replace the evaluation process with our quantization-aware accuracy predictor to estimate the performance of each candidate directly. The cost for each candidate can then be reduced from N times of model inference to only one time of predictor inference (where N is the size of the validation set). Furthermore, we can verify the resource constraints by our latency/energy lookup table to avoid the direct interaction with the target hardware. Given a resource budget, we directly eliminate the candidates that exceed the constraints. Table 2: Comparison with state-of-the-art efficient models for hardware with fixed quantization or mixed precision. Our method cuts down the marginal search time by two-order of magnitudes while achieving better performance than others. The marginal CO 2 emission (lbs) and cloud compute cost ($) is negligible for search in a new scenario. Data Preparation for Quantization-aware Accuracy Predictor. We generate two kinds of data (2,500 for each): 1. random sample both architecture and quantization policy; 2. random sample architecture, and sample 10 quantization policies for each architecture configuration. To speed up the data collection process, we use ImageNet-100 dataset. We mix the data for training the quantizationaware accuracy predictor, and use full-precision pretrained predictor's weights to transfer. The number of data to train a full precision predictor is 80,000. In that way, our quantization accuracy predictor can have the ability to generalize among different architecture/quantization policy pairs and learn the mutual relation between architecture and quantization policy. Evolutionary Architecture Search. For evolutionary architecture search, we set the population size to be 100, and choose Top-25 candidates to produce the next generation (50 by mutation, 50 by crossover). The mutation rate is 0.1, which is the same as that in . We set max iterations to 500, and choose the best candidate among the final population. Quantization. We follow the implementation in to do quantization. Specifically, we quantize the weights and activations with the specific quantization policies. For each layer with weights w with quantization bit b, we linearly quantize it to [−v, v], the quantized weight is: We set choose different v for each layer that minimize the KL-divergence D(w||w) between origin weights w and quantized weights w. For activation weights, we quantize it to [0, v] since the value is non-negative after ReLU6 layer. To verify the effectiveness of our methods, we conduct experiments that cover two of the most important constraints for on-device deployment: latency and energy consumption in comparison with some state-of-the-art models using neural architecture search. Besides, we compare BitOps with some multi-stage optimized models. Dataset, Models and Hardware Platform. The experiments are conducted on ImageNet dataset. We compare the performance of our end-to-end designed models with mixed-precision models searched by; b) and some SOTA fixed precision 8-bit models. The platform we used to measure the resource consumption for mixed-precision model is BitFusion , which is a state-of-the-art spatial ASIC design for neural network accelerator. It employs a 2D systolic array of Fusion Units which spatially sum the shifted partial products of two-bit elements from weights and activations. Figure 3: Comparison with mixed-precision models searched by HAQ under latency/energy constraints. When the constraint is strict, our model can outperform fixed precision model by more than 10% accuracy, and 5% compared with HAQ. Such performance boost may benefit from the dynamic architecture search space rather than fixed one as MobileNetV2. Figure 4: Comparison with sequentially designed mixed-precision models searched by AMC and HAQ (b; ; under latency constraints. Our end-to-end designed model while achieving better accuracy than sequentially designed models. 75.1 74.6 +0.5% Acc with 2.2x BitOps saving Table 2 presents the for different efficiency constraints. As one can see, our model can consistently outperform state-of-the-art models with either fixed or mixed-precision. Specifically, our small model (Ours-B) can have 2.2% accuracy boost than mixed-precision MobileNetV2 search by HAQ (from 71.9% to 74.1%); our large model (Ours-C) attains better accuracy (from 74.6% to 75.1%) while only requires half of BitOps. When applied with transfer technology, it does help for the model to get better performance (from 72.1% to 74.1%). It is also notable that the marginal cost for cloud computer and CO 2 emission is two orders of magnitudes smaller than other works. Comparison with MobileNetV2+HAQ. Figure 3 show the on the BitFusion platform under different latency constraints and energy constraints. Our end-to-end designed models consistently outperform both mixed-precision and fixed precision SOTA models under certain constraints. It is notable when constraint is tight, our models have significant improvement compared with stateof-the-art mixed-precision models. Specifically, with similar efficiency constraints, we improve the ImageNet top1 accuracy from the MobileNetV2 baseline 61.4% to 71.9% (+10.5%) and 72.7% (+11.3%) for latency and energy constraints, respectively. Moreover, we show some models searched by our quantization-aware predictor without predictor-transfer technique. With this technique applied, Right graph shows that when data is limited, predictor-transfer technique could largely improve the pairwise accuracy (from 64.6% to 75.6%). Using predictor-transfer technique, we can achieve 85% pairwise accuracy using less than 3k data points, while at least 4k data will be required without this technique. the accuracy can consistently have an improvement, since the non-transferred predictor might loss some mutual information between architecture and quantization policy. Comparison with multi-stage optimized Model. Figure 4 compares the multi-stage optimization with our joint optimization . As one can see, under the same latency/energy constraint, our model can attain better accuracy than the multi-stage optimized model (74.1% vs 71.8%). This is reasonable since the per-stage optimization might not find the global optimal model as end-to-end design does. Comparison under Limited BitOps. Figure 5 reports the with limited BitOps budget. As one can see, under a tight BitOps constraint, our model improves over 2% accuracy (from 71.5% to 73.9%) compared with searched model using . Moreover, our models achieve the same level accuracy (75.1%) as ResNet34 full precision model while only consumes half of the BitOps as 4-bit version (from 52.83G to 25.69G). Figure 6 shows the performance of our predictor-transfer technique compared with training from scratch. For each setting, we train the predictor to convergence and evaluate the pairwise accuracy (i.e. the proportion that predictor correctly identifies which is better between two randomly selected candidates from a held-out dataset), which is a measurement for the predictor's performance. As shown, the transferred predictor have a higher and faster pairwise accuracy convergence. Also, when the data is very limited, our method can have more than 10% pairwise accuracy over scratch training. We propose EMS, an end-to-end design method for architecting mixed-precision model. Unlike former works that decouple into separated stages, we directly search for the optimal mixed-precision architecture without multi-stage optimization. We use predictor-base method that can have no extra evaluation for target dataset, which greatly saves GPU hours for searching under an upcoming scenario, thus reducing marginally CO 2 emission and cloud compute cost. To tackle the problem for high expense of data collection, we propose predictor-transfer technique to make up for the limitation of data. Comparisons with state-of-the-art models show the necessity of joint optimization and prosperity of our end-to-end design method."

We present a software framework for transforming distributions and demonstrate its flexibility on relaxing mean-field assumptions in variational inference with the use of coupling flows to replicate structure from the target generative model.

We use continuous time dynamics to define a generative model with exact likelihoods and efficient sampling that is parameterized by unrestricted neural networks.


Question: Represent the scientific article to find a short TLDR

Question: Multi-domain learning (MDL) aims at obtaining a model with minimal average risk across multiple domains. Our empirical motivation is automated microscopy data, where cultured cells are imaged after being exposed to known and unknown chemical perturbations, and each dataset displays significant experimental bias. This paper presents a multi-domain adversarial learning approach, MuLANN, to leverage multiple datasets with overlapping but distinct class sets, in a semi-supervised setting. Our contributions include: i) a bound on the average- and worst-domain risk in MDL, obtained using the H-divergence; ii) a new loss to accommodate semi-supervised multi-domain learning and domain adaptation; iii) the experimental validation of the approach, improving on the state of the art on two standard image benchmarks, and a novel bioimage dataset, Cell. Advances in technology have enabled large scale dataset generation by life sciences laboratories. These datasets contain information about overlapping but non-identical known and unknown experimental conditions. A challenge is how to best leverage information across multiple datasets on the same subject, and to make discoveries that could not have been obtained from any individual dataset alone. Transfer learning provides a formal framework for addressing this challenge, particularly crucial in cases where data acquisition is expensive and heavily impacted by experimental settings. One such field is automated microscopy, which can capture thousands of images of cultured cells after exposure to different experimental perturbations (e.g from chemical or genetic sources). A goal is to classify mechanisms by which perturbations affect cellular processes based on the similarity of cell images. In principle, it should be possible to tackle microscopy image classification as yet another visual object recognition task. However, two major challenges arise compared to mainstream visual object recognition problems BID51. First, biological images are heavily impacted by experimental choices, such as microscope settings and experimental reagents. Second, there is no standardized set of labeled perturbations, and datasets often contain labeled examples for a subset of possible classes only. This has limited microscopy image classification to single datasets and does not leverage the growing number of datasets collected by the life sciences community. These challenges make it desirable to learn models across many microscopy datasets, that achieve both good robustness w.r.t. experimental settings and good class coverage, all the while being robust to the fact that datasets contain samples from overlapping but distinct class sets. Multi-domain learning (MDL) aims to learn a model of minimal risk from datasets drawn from distinct underlying distributions BID20, and is a particular case of transfer learning BID46. As such, it contrasts with the so-called domain adaptation (DA) problem BID7 BID5 BID22 BID46. DA aims at learning a model with minimal risk on a distribution called "target" by leveraging other distributions called "sources". Notably, most DA methods assume that target classes are identical to source classes, or a subset thereof in the case of partial DA BID77.The expected benefits of MDL, compared to training a separate model on each individual dataset, are two-fold. First, MDL leverages more (labeled and unlabeled) information, allowing better generalization while accommodating the specifics of each domain BID20 BID72. Thus, MDL models have a higher chance of ab initio performing well on a new domain − a problem referred to as domain generalization BID44 or zero-shot domain adaptation BID74. Second, MDL enables knowledge transfer between domains: in unsupervised and semi-supervised settings, concepts learned on one domain are applied to another, significantly reducing the need for labeled examples from the latter BID46. Learning a single model from samples drawn from n distributions raises the question of available learning guarantees regarding the model error on each distribution. BID32 introduced the notion of H-divergence to measure the distance between source and target marginal distributions in DA. BID4 have shown that a finite sample estimate of this divergence can be used to bound the target risk of the learned model. The contributions of our work are threefold. First, we extend the DA guarantees to MDL (Sec. 3.1), showing that the risk of the learned model over all considered domains is upper bounded by the oracle risk and the sum of the H-divergences between any two domains. Furthermore, an upper bound on the classifier imbalance (the difference between the individual domain risk, and the average risk over all domains) is obtained, thus bounding the worst-domain risk. Second, we propose the approach Multi-domain Learning Adversarial Neural Network (MULANN), which extends Domain Adversarial Neural Networks (DANNs) BID22 to semi-supervised DA and MDL. Relaxing the DA assumption, MULANN handles the so-called class asymmetry issue (when each domain may contain varying numbers of labeled and unlabeled examples of a subset of all possible classes), through designing a new loss (Sec. 3.2). Finally, MULANN is empirically validated in both DA and MDL settings (Sec. 4), as it significantly outperforms the state of the art on three standard image benchmarks BID52 BID35, and a novel bioimage benchmark, CELL, where the state of the art involves extensive domain-dependent pre-processing. Notation. Let X denote an input space and Y = {1, . . ., L} a set of classes. For i = 1,..., n, dataset S i is an iid sample drawn from distribution D i on X × Y. The marginal distribution of D i on X is denoted by D X i. Let H be a hypothesis space; for each h in H (h : X → Y) we define the risk under distribution D i as i (h) = P x,y∼Di (h(x) = y). h i (respectively h) denotes the oracle hypothesis according to distribution D i (resp. with minimal total risk over all domains): DISPLAYFORM0 In the semi-supervised setting, the label associated with an instance might be missing. In the following, "domain" and "distribution" will be used interchangeably, and the "classes of a domain" denote the classes for which labeled or unlabeled examples are available in this domain. Machine learning classically relies on the iid setting: when training and test samples are independently drawn from the same joint distribution P (X, Y) BID71. Two other settings emerged in the 1990s, "concept drift" and "covariate shift". They respectively occur when conditional data distributions P (Y |X) and marginal data distributions P (X) change, either continuously or abruptly, across training data or between train and test data BID56. Since then, transfer learning has come to designate methods to learn across drifting, shifting or distinct distributions, or even distinct tasks (; BID46 In MDL, the different domains can be taken into account by maintaining shared and domain-specific parameters BID20, or through a domain-specific use of shared parameters. The domaindependent use of these parameters can be learned, e.g. using domain-guided dropout BID72, or based on prior knowledge about domain semantic relationships BID74 .Early DA approaches leverage source examples to learn on the target domain in various ways, e.g. through reweighting source datapoints BID41 BID26 BID24, or defining an extended representation to learn from both source and target BID19 . Other approaches proceed by aligning the source and target representations with PCA-based correlation alignment, or subspace alignment BID21 . In the field of computer vision, a somewhat related way of mapping examples in one domain onto the other is image-to-image translation, possibly in combination with a generative adversarial network (see references in Appendix A).Intuitively, the difficulty of DA crucially depends on the distance between source and target distribution. Accordingly, a large set of DA methods proceed by reducing this distance in the original input space X, e.g. via importance sampling BID7 or by modifying the source representation using optimal transport BID17 BID18. Another option is to map source and target samples on a latent space where they will have minimal distance. Neural networks have been intensively exploited to build such latent spaces, either through generative adversarial mechanisms BID67 BID23, or through combining task objective with an approximation of the distance between source(s) and target. Examples of used distances include the Maximum Mean Discrepancy due to BID25 BID66 BID10, some of its variants BID38, the L 2 contrastive divergence BID43, the Frobenius norm of the output feature correlation matrices, or the H-divergence BID4 BID22 BID47 BID40 ) (more in Sec. 3). Most DA methods assume that source(s) and target contain examples from the same classes; in particular, in standard benchmarks such as OFFICE BID52, all domains contain examples from the same classes. Notable exceptions are partial DA methods, where target classes are expected to be a subset of source classes e.g. BID77. DA and partial DA methods share two drawbacks when applied to semi-supervised MDL with non-identical domain class sets. First, neither generic nor partial DA methods try to mitigate the impact of unlabeled samples from a class without any labeled counterparts. Second, as they focus on target performance, (partial) DA methods do not discuss the impact of extra labeled source classes on source accuracy. However, as shown in Sec. 4.3, class asymmetry can heavily impact model performance if not accounted for. Bioinformatics is increasingly appreciating the need for domain adaptation methods BID55 BID73 BID68. Indeed, experimentalists regularly face the issues of concept drift and covariate shift. Most biological experiments that last more than a few days are subject to technical variations between groups of samples, referred to as batch effects. Batch effects in image-based screening data are usually tackled with specific normalization methods BID8 ). More recently, work by applied CorAl for this purpose, aligning each batch with the entire experiment. DA has been applied to image-based datasets for improving or accelerating image segmentation tasks BID3 BID70 BID6 BID30. However, to our knowledge, MDL has not yet been used in Bioimage Informatics, and this work is the first to leverage distinct microscopy screening datasets using MDL. The H-divergence has been introduced to bound the DA risk BID4 BID22. This section extends the DA theoretical to the MDL case (Sec. 3.1), supporting the design of the MULANN approach (Sec. 3.2). The reader is referred to Appendix B for formal definitions and proofs. The distance between source and target partly governs the difficulty of DA. The H-divergence has been introduced to define such a distance which can be empirically estimated with proven guarantees BID2 BID32. This divergence measures how well one can discriminate between samples from two marginals. It inspired an adversarial approach to DA BID22, through the finding of a feature space in which a binary classification loss between source and target projections is maximal, and thus their H-divergence minimal. Furthermore, the target risk is upper-bounded by the empirical source risk, the empirical H-divergence between source(s) and target marginals, and the oracle DA risk BID4 BID76.Bounding the MDL loss using the H-divergence. A main difference between DA and MDL is that MDL aims to minimize the average risk over all domains while DA aims to minimize the target risk only. Considering for simplicity a binary classification MDL problem and taking inspiration from BID42 BID5, the MDL loss can be formulated as an optimal convex combination of domain risks. A straightforward extension of Ben-David et al. FORMULA0 (Theorem 2 in Appendix B.2) establishes that the compound empirical risk is upper bounded by the sum of: i) the oracle risk on each domain; ii) a statistical learning term involving the VC dimension of H; iii) the divergence among any two domains as measured by their H-divergence and summed oracle risk. This states that, assuming a representation in which domains are as indistinguishable as possible and on which every 1-and 2-domain classification task is well addressed, then there exists a model that performs well on all of them. In the 2-domain case, the bound is minimized when one minimizes the convex combination of losses in the same proportion as samples. Bounding the worst risk. The classifier imbalance w.r.t. the i-th domain is defined as DISPLAYFORM0 The extent to which marginal D i can best be distinguished by a classifier from H (i.e., the Hdivergence), and the intrinsic difficulty i of the i-th classification task, yield an upper-bound on the classifier imbalance (proof in Appendix B.3): Proposition 1. Given an input space X, n distributions D i over X × {0, 1} and hypothesis class H on X, for any h ∈ H, let i (h) (respectively¯ (h)) denote the classification risk of h w.r.t. distribution D i (resp. its average risk over all D i). The risk imbalance | i (h) −¯ (h)| is upper bounded as: DISPLAYFORM1 Accordingly, every care taken to minimize H-divergences or ∆ ij (e.g. using the class-wise contrastive losses BID43) improves the above upper bound. An alternative bound of the classifier imbalance can be obtained by using the H∆H-divergence (proposition 3, and corollaries 4, 5 for the 2-domain case in Appendix). As pointed out by e.g. BID47, when minimizing the H-divergence between two domains, a negative transfer can occur in the case of class asymmetry, when domains involve distinct sets of classes. For instance, if a domain has unlabeled samples from a class which is not present in the other domains, both global BID22 and class-wise BID47 ) domain alignments will likely deteriorate at least one of the domain risks by putting the unlabeled samples close to labeled ones from the same domain. A similar issue arises if a domain has no (labeled or unlabeled) samples in classes which are represented in other domains. In general, unlabeled samples are only subject to constraints from the domain discriminator, as opposed to labeled samples. Thus, in the case of class asymmetry, domain alignment will tend to shuffle unlabeled samples more than labeled ones. This limitation is addressed in MULANN by defining a new discrimination task referred to as Known Unknown Discrimination (KUD). Let us assume that, in each domain, a fraction p of unlabeled samples comes from extra classes, i.e. classes with no labeled samples within the domain. KUD aims at discriminating, within each domain, labeled samples from unlabeled ones that most likely belong to such extra classes. More precisely, unlabeled samples of each domain are ranked according to the entropy of their classification according to the current classifier, restricted to their domain classes. Introducing the hyper-parameter p, the top p% examples according to this classification entropy are deemed "most likely unknown", and thus discriminated from the labeled ones of the same domain. The KUD module aims at repulsing the most likely unknown unlabeled samples from the labeled ones within each domain (FIG1, thus resisting the contractive effects of global domain alignment. DISPLAYFORM0 Overall, MULANN involves 3+n interacting modules, where n is the number of domains with unlabeled data. The first module is the feature extractor with parameters θ f, which maps the input space X to some latent feature space Ω. 2+n modules are defined on Ω: the classifier module, the domain discriminator module, and the n KUD modules, with respective parameters θ c, θ d and (θ u,i) i. All modules are simultaneously learned by minimizing loss DISPLAYFORM1 where ζ and λ are hyper-parameters, DISPLAYFORM2 is the domain discrimination loss (multi-class cross-entropy loss of classifying examples from S i in class i), and L i u (θ f, θ u,i) is the KUD loss (binary cross-entropy loss of discriminating labelled samples from S i from the "most likely unknown" unlabelled samples from S i).The loss minimization aims to find a saddle point (θ f,θ y,θ d,θ u), achieving an equilibrium between the classification performance, the discrimination among domains (to be prevented) and the discrimination among labeled and some unlabeled samples within each domain (to be optimized). The sensitivity w.r.t. hyperparameter p will be discussed in Sec. 4.3. This section reports on the experimental validation of MULANN in DA and MDL settings on three image datasets (Sec. 4.2), prior to analyzing MULANN and investigating the impact of class asymmetry on model performances (Sec. 4.3). Datasets The DA setting considers three benchmarks: DIGITS, including the well-known MNIST and MNIST-M BID35 BID22; Synthetic road signs and German traffic sign benchmark BID14 BID60 and OFFICE BID52. The MDL setting considers the new CELL benchmark, which is made of fluorescence microscopy images of cells (detailed in Appendix C). Each image contains tens to hundreds of cells that have been exposed to a given chemical compound, in three domains: California (C), Texas (T) and England (E). There are 13 classes across the three domains (Appendix FIG5 ; a drug class is a group of compounds targeting a similar known biological process, e.g. DNA replication. Four domain shifts are considered: C↔T, T↔E, E↔C and C↔T↔E.Baselines and hyperparameters. In all experiments, MULANN is compared to DANN BID22 and its extension MADA BID47 (that involves one domain discriminator module per class rather than a single global one). For DANN, MADA and MULANN, the same pre-trained VGG-16 architecture BID59 from Caffe BID28 ) is used for OFFICE and CELL 2; the same small convolutional network as BID22 is used for DIGITS (see Appendix D.1 for details). The models are trained in Torch BID16 using stochastic gradient descent with momentum (ρ = 0.9). As in BID22, no hyper-parameter grid-search is performed for OFFICE -double cross-validation is used for all other benchmarks. Hyper-parameter ranges can be found in Appendix D.2.Semi-supervised setting. For OFFICE and CELL, we follow the experimental settings from BID52. A fixed number of labeled images per class is used for one of the domains in all cases (20 for Amazon, 8 for DSLR and Webcam, 10 in CELL). For the other domain, 10 labeled images per class are used for half of the classes (15 for OFFICE, 4 for CELL). For DIGITS and RoadSigns, all labeled source train data is used, whereas labeled target data is used for half of the classes only (5 for DIGITS, 22 for RoadSigns). In DA, the evaluation is performed on all target images from the unlabeled classes. In MDL, the evaluation is performed on all source and target classes (considering labeled and unlabeled samples).Evaluation goals. A first goal is to assess MULANN performance comparatively to the baselines. A second goal is to assess how the experimental setting impacts model performance. As domain discriminator and KUD modules can use both labeled and unlabeled images, a major question regards the impact of seeing unlabeled images during training. Two experiments are conducted to assess this impact: a) the same unlabeled images are used for training and evaluation (referred to as fully transductive setting, noted FT); b) some unlabeled images are used for training, and others for evaluation (referred to as non-fully transductive setting, noted NFT). (The case where no unlabeled images are used during training is discarded due to poor ). DA on DIGITS, RoadSigns and OFFICE. BID43 ) (legend CCSA), that uses a contrastive loss to penalizes large (resp. small) distances between same (resp. different) classes and different domains in the feature space; Published from BID65, an extension of DANN that adds a loss on target softmax values ("soft label loss"; legend Tseng15). Overall, MULANN yields the best , significantly improving upon the former best on the most difficult cases, i.e., D→A, A→D or W→A. As could be expected, the fully transductive match or significantly outperform the non-fully transductive ones. Notably, MADA performs similarly to DANN on DIGITS and RoadSigns, but worse on OFFICE; a potential explanation is that MADA is hindered as the number of classes, and thus domain discriminators, increases (respectively 10, 32 and 43 classes).MDL on CELL. A state of the art method for fluorescence microscopy images relies on tailored approaches for quantifying changes to cell morphology BID31. Objects (cells) are segmented in each image, and circa 650 shape, intensity and texture features are extracted for each object in each image. The profile of each image is defined as the vector of its Kolmogorov-Smirnov statistics, computed for each feature by comparing its distribution to that of the same feature from pooled negative controls of the same plate 3. Classification in profile space is realized using linear discriminant analysis, followed by k-nearest neighbor (LDA+k-NN) ("Baseline P" in Table 2). As a state of the art shallow approach to MDL to be applied in profile space, CORAL was chosen ("P + CORAL" in Table 2). A third baseline corresponds to fine-tuning VGG-16 without any transfer loss ("Baseline NN"). Table 2 compares DANN, MADA and MULANN to the baselines, where columns 4-7 (resp. 8-9) consider raw images (resp. the profile representations). 4 The fact that a profile-based baseline generally outperforms an image-based baseline was expected, as profiles are designed to reduce the impact of experimental settings (column 4 vs. 8). The fact that standard deviations tend to be larger Table 2: CELL test classification accuracy on all domains (average and stdev on 5 folds), in the fully transductive setting (see TAB10 in Appendix for non-transductive ones, and sections C.4, C.5 for details about image and class selection).Shift Image set # classes Baseline NN DANN MADA MULANN Baseline P P+Coral E-C E 7 63.7 (7.0) 62.9 (7.6) 59.5 (9.5) 64.4 (8.0) 74.1 (3.9) 58.4 (6.1) C lab. here than for OFFICE, RoadSigns or DIGITS is explained by a higher intra-class heterogeneity; some classes comprise images from different compounds with similar but not identical biological activity. Most interestingly, MULANN and P+CORAL both improve classification accuracy on unlabeled classes at the cost of a slighty worse classification accuracy for the labeled classes (in all cases but one). This is explained as reducing the divergence between domain marginals on the latent feature space prevents the classifier from exploiting dataset-dependent biases. Overall, MULANN and P+CORAL attain comparable on two-domain cases, with MULANN performing significantly better in the three-domain case. Finally, MULANN matches or significantly outperforms DANN and MADA. Sensitivity w.r.t. the fraction p of "known unknowns". MULANN was designed to counter the negative transfer that is potentially caused by class asymmetry. This is achieved through the repulsion of labeled examples in each domain from the fraction p of unlabeled examples deemed to belong to extra classes (not represented in the domain). The sensitivity of MULANN performance to the value of p and its difference to the ground truth p is investigated on MNIST↔MNIST-M. A first remark is that discrepancies between p and p has no influence on the accuracy on a domain without unlabeled datapoints (Fig. 4 in Appendix). FIG1, right, displays the error depending on p for various values of p. As could have been expected, it is better to underestimate than to overestimate p; it is even better to slightly underestimate it than to get it right, as the entropy ranking of unlabeled examples can be perturbed by classifier errors. Impact of class/domain asymmetry. Section 4.2 reports on the classification accuracy when all classes are represented in all domains of a given shift. In the general case however, the classes represented by the unlabeled examples are unknown, hence there might exist "orphan" classes, with labeled or unlabeled samples, unique to a single domain. The impact of such orphan classes, referred to as class asymmetry, is investigated in the 2-domain case. Four types of samples are considered TAB5: A class might have labeled examples in both domains (α), labeled in one domain and unlabeled in the other domain (β), labeled in one domain and absent in the other one (orphan γ), and finally unlabeled in one domain and absent in the other one (orphan δ). The impact of the class asymmetry is displayed on FIG3, reporting the average classification accuracy of α, β classes on domain 1 on the x-axis, and classification accuracy of unlabeled β classes on domain 2 on the y-axis, for MULANN, DANN and MADA on OFFICE (on CELL in Fig. 5, Appendix).A clear trend is that adding labeled orphans γ (case "2", FIG3) entails a loss of accuracy for all algorithms compared to the no-orphan reference (case "1"). This is explained as follows: on the one hand, the γ samples are subject to the classifier pressure as all labeled samples; on the other hand, they must be shuffled with samples from domain 2 due to the domain discriminator(s) pressure. Thus, the easiest solution is to shuffle the unlabeled β samples around, and the loss of accuracy on these β samples is very significant (the "2" is lower on the y-axis compared to "1" for all algorithms). The perturbation is less severe for the labeled (α, β) samples in domain 1, which are preserved by the classifier pressure (x-axis).The in case "3" are consistent with the above explanation: since the unlabeled δ samples are only seen by the discriminator(s), their addition has little impact on either the labeled or unlabeled data classification accuracy FIG3. Finally, there is no clear trend in the impact of both labeled and unlabeled orphans (case "4"): labeled (α, β) (resp. unlabeled β) are only affected for MADA on CELL (resp. MULANN on OFFICE). Overall, these show that class asymmetry matters for practical applications of transfer learning, and can adversely affect all three adversarial methods FIG3, with asymmetry in labeled class content ("2") being the most detrimental to model performance. This paper extends the use of domain adversarial learning to multi-domain learning, establishing how the H-divergence can be used to bound both the risk across all domains and the worst-domain risk (imbalance on a specific domain). The stress is put on the notion of class asymmetry, that is, when some domains contain labeled or unlabeled examples of classes not present in other domains. Showing the significant impact of class asymmetry on the state of the art, this paper also introduces MULANN, where a new loss is meant to resist the contractive effects of the adversarial domain discriminator and to repulse (a fraction of) unlabeled examples from labeled ones in each domain. The merits of the approach are satisfactorily demonstrated by comparison to DANN and MADA on DIGITS, RoadSigns and OFFICE, and obtained on the real-world CELL problem establish a new baseline for the microscopy image community. A perspective for further study is to bridge the gap between the proposed loss and importance sampling techniques, iteratively exploiting the latent representation to identify orphan samples and adapt the loss while learning. Further work will also focus on how to identify and preserve relevant domain-specific behaviours while learning in a domain adversarial setting (e.g., if different cell types have distinct responses to the same class of perturbations). This work was supported by NIH RO1 CA184984 (LFW), R01GM112690 (SJA) and the Institute of Computational Health Sciences at UCSF (SJA and LFW). We thank the Shoichet lab (UCSF) for access to their GPUs and Theresa Gebert for suggestions and feedback. In the field of computer vision, another way of mapping examples in one domain onto the other domain is image-to-image translation. In the supervised case (the true pairs made of an image and its translation are given), Pic2Pix trains a conditional GAN to discriminate true pairs from fake ones. In the unsupervised case, another loss is designed to enforce cycle consistency (simultaneously learning the mapping φ from domain A to B, ψ from B to A, and requiring φoψ =Id) BID75. Note that translation approaches do not per se address domain adaptation as they are agnostic w.r.t. the classes. Additional losses are used to overcome this limitation: Domain transfer network (DTN) BID64 uses an auto-encoder-like loss in the latent space; GenToAdapt BID53 ) uses a classifier loss in the latent space; UNIT BID36 ) uses a VAE loss. StarGAN BID15 combines image-to-image translation with a GAN, where the discriminator is trained to discriminate true from fake pairs on the one hand, and the domain on the other hand. ComboGAN BID1 learns two networks per domain, an encoder and a decoder. DIRT-T BID57 ) uses a conditional GAN and a classifier in the latent space, with two additional losses, respectively enforcing the cluster assumption (the classifier boundary should not cross high density region) and a virtual adversarial training (the hypothesis should be invariant under slight perturbations of the input). Interestingly, DA and MDL (like deep learning in general) tend to combine quite some losses; two benefits are expected from using a mixture of losses, a smoother optimization landscape and a good stability of the representation BID11. Definition. BID32 BID4 Given a domain X, two distributions D and D over that domain and a binary hypothesis class H on X, the H-divergence between D and D is defined as: DISPLAYFORM0 Theorem 2. Given an input space X, we consider n distributions D i over X ×{0; 1} and a hypothesis class H on X of VC dimension d. Let α and γ be in the simplex of dimension n. If S is a sample of size m which contains γ i m samples from D i, andĥ is the empirical minimizer of i α iˆ i on (S i) i, then for any δ > 0, with probability at least 1 − δ, the compound empirical error is upper bounded as: DISPLAYFORM0 with DISPLAYFORM1 A tighter bound can be obtained by BID5 operates on the symmetric difference hypothesis space H∆H. However, divergence H∆H does not lend itself to empirical estimation: even BID5 fall back on H-divergence in their empirical validation. DISPLAYFORM2 the n-dimensional simplex and h ∈ H, we note α (h) = i α i i (h).We have for α in the simplex of dimension n, h ∈ H and j ∈ {1, . . ., m}, using the triangle inequality (similarly to the proof of Theorem 4 in BID5) DISPLAYFORM0 The last line follows from the definitions of β i,j and H-divergence. Thus using lemma 6 in DISPLAYFORM1 Hence the . Proof of proposition 1 We have for h ∈ H and j ∈ [1, . . ., m], using the triangle inequality and the definition of i (similarly to the proof of Theorem 1 in BID4) DISPLAYFORM0 We have for i DISPLAYFORM1 The second line follows from the triangle inequality and the definition of the H-divergence. Thus DISPLAYFORM2 By symmetry we obtain 1 n DISPLAYFORM3 Thus the . Proposition 3. Given a domain X, m distributions D i over X × {0; 1} and a hypothesis class H on X, we have for h ∈ H and j ∈ [1, . . ., m] DISPLAYFORM4 The second line follows from Lemma 3 from BID5, and the third from the triangle inequality. From this and proposition 1 we obtain the . Corollaries for the 2-domain case Corollary 4. Given a domain X, two distributions D S and D T over X × {0, 1} and a hypothesis class H on X, we have for DISPLAYFORM5 Corollary 5. Given a domain X, two distributions D S and D T over X × {0; 1} and a hypothesis class H on X, we have for DISPLAYFORM6 This dataset is extracted from that published in BID31. It contains 455 biologically active images, in 11 classes, on four 384-well plates, in three channels: H2B-CFP, XRCC5-YFP and cytoplasmic-mCherry. Our analysis used 10 classes:'Actin','Aurora','DNA','ER','HDAC','Hsp90','MT','PLK','Proteasome','mTOR'.On top of the quality control from the original paper, a visual quality control was implemented to remove images with only apoptotic cells, and XRCC5-YFP channel images were smoothed using a median filter of size 2 using SciPy BID29. This dataset is designed to be similar to the Texas domain BID31, generated using the same cell line, but in a different laboratory, by a different biologist, and using different equipment. It contains 1,077 biologically active images, in 10 classes, on ten 384-well plates, in three channels: H2B-CFP, XRCC5-YFP and cytoplasmic-mCherry. The classes are:'Actin','Aurora','DNA','ER','HDAC','Hsp90','MT','PLK','Proteasome','mTOR'.Cell culture, drug screening and image acquisition Previously BID31, retroviral transduction of a marker plasmid "pSeg" was used to stably express H2B-CFP and cytoplasmicmCherry tags in A549 human lung adenocarcinoma cells. A CD-tagging approach BID58 was used to add an N-terminal YFP tag to endogenous XRCC5.Cells were maintained in RPMI1640 media containing 10% FBS, 2 mM glutamine, 50 units/ml penicillin, and 50 µg/ml streptomycin (all from Life Technologies, Inc.), at 37 • C, 5% CO 2 and 100% humidity. 24h prior to drug addition, cells were seeded onto 384-well plate at a density of 1200 cells/well. Following compound addition, cells were incubated at 37• C for 48 hours. Images were then acquired using a GE InCell Analyzer 2000. One image was acquired per well using a 10x objective lens with 2x2 binning. Image processing Uneven illumination was corrected as described in BID61. Background noise was removed using the ImageJ RollingBall plugin BID54. Images were segmented, object features extracted and biological activity determined as previously described BID31. A visual quality control was implemented to remove images with obvious anomalies (e.g. presence of a hair or out-of-focus image) and images with only apoptotic cells. YFP-XRCC5 channel images were smoothed using a median filter of size 2. This dataset was published by BID12 and retrieved from BID37. It contains 879 biologically active images of MCF7 breast adenocarcinoma cells, in 15 classes on 55 96-well plates, in 3 channels: Alexa Fluor 488 (Tubulin), Alexa Fluor 568 (Actin) and DAPI (nuclei). Classes with fewer than 15 images and absent from the other datasets ("Calcium regulation", "Cholesterol", "Epithelial", "MEK", "mTOR") were not used, which leaves 10 classes:'Actin','Aurora','DNA','ER','Eg5 inhibitor','HDAC','Kinase','MT','Proteasome','Protein synthesis'.Image processing As the images were acquired using a 20X objective, they were stitched using ImageJ plugin BID50 ) and down-scaled 2 times. Cells thus appear the same size as in the other domains. Images were segmented, object features extracted and biological activity obtained as previously described BID31. A visual quality control was implemented to remove images with obvious anomalies and images with only apoptotic cells. Images with too few cells were also removed: an Otsu filter BID45 was used to estimate the percentage of pixels containing nuclei in each image, and images with less than 1% nuclear pixels were removed. Tubulin channel images were smoothed using a median filter of size 2. Images which were not significantly distinct from negative controls were identified as previously BID31 and excluded from our analysis. Previous work on the England dataset further focused on images which "clearly [have] one of 12 different primary mechanims of action" BID37. We chose not to do so, since it in a simpler problem (90% accuracy easy to reach) with much less room for improvement. Images from all domains were down-scaled 4 times and flattened to form RGB images. Images were normalized by subtracting the intensity values from negative controls (DMSO) of the same plate in each channel. England, Texas and California share images for cell nucleus and cytoplasm, but their third channel differs: Texas and California shows the protein XRCC5, whereas England shows the Actin protein. Therefore, the experiments which combine Texas and England, and California and England used only the first two channels, feeding an empty third channel into the network. Similarly, profiles contain 443 features which are related to the first two channels, and 202 features which are related to the third channel. Only the former were used in experiments which involve the England dataset. Shift Dom. 2, labeled classes Domain 2, unlabeled classes E-C HDAC, Proteasome, Actin, Aurora DNA, MT, ER C-T DNA, HDAC, MT, ER, Aurora, mTOR, PLK Actin, Proteasome, Hsp90 T-E DNA, MT, Proteasome, Actin, ER Aurora, HDAC, Actin C-T-E DNA, MT, Proteasome, Actin, ER Aurora, HDAC, Actin BID22 BID66, a bottleneck fully connected layer is added after the last dense layer of VGG-16. Learning rates on weights (resp. biases) from "from scratch" layers is ten (resp, twenty) times that on parameters of fine-tuned layers. Instance normalization is used on DIGITS, whereas global normalization is used on OFFICE and CELL. Exponentially increasing, constant ζ 0.1, 0.8 TAB10: Range of hyper-parameters which were evaluated in cross-validation experiments. Exponentially decreasing schedule, exponentially increasing schedule, indiv. lr (learning rates from layers which were trained from scratch are multiplied by 10), as in BID22 ).E ADDITIONAL E.1 3-DOMAIN ON OFFICE We use tSNE BID69 to visualize the common feature space in the example of Webcam → Amazon. FIG3 shows that classes are overall better separated with MULANN. In particular, when using MULANN, unlabeled examples (blue) are both more grouped and closer to labeled points from the other domain. E.3 SEMI-SUPERVISED MDL ON THE BIO DATASET

Represent the short summary for finding the corresponding scientific article (long)

Adversarial Domain adaptation and Multi-domain learning: a new loss to handle multi- and single-domain classes in the semi-supervised setting.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.