Question: represent the text to find a very short summary (1-2 sentences max).

Question: Exploration while learning representations is one of the main challenges Deep Reinforcement Learning (DRL) faces today. As the learned representation is dependant in the observed data, the exploration strategy has a crucial role. The popular DQN algorithm has improved significantly the capabilities of Reinforcement Learning (RL) algorithms to learn state representations from raw data, yet, it uses a naive exploration strategy which is statistically inefficient. The Randomized Least Squares Value Iteration (RLSVI) algorithm , on the other hand, explores and generalizes efficiently via linearly parameterized value functions. However, it is based on hand-designed state representation that requires prior engineering work for every environment. In this paper, we propose a Deep Learning adaptation for RLSVI. Rather than using hand-design state representation, we use a state representation that is being learned directly from the data by a DQN agent. As the representation is being optimized during the learning process, a key component for the suggested method is a likelihood matching mechanism, which adapts to the changing representations. We demonstrate the importance of the various properties of our algorithm on a toy problem and show that our method outperforms DQN in five Atari benchmarks, reaching competitive with the Rainbow algorithm. In Reinforcement Learning (RL), an agent seeks to maximize the cumulative rewards obtained from interactions with an unknown environment . Since the agent can learn only by its interactions with the environment, it faces the exploration-exploitation dilemma: Should it take actions that will maximize the rewards based on its current knowledge or instead take actions to potentially improve its knowledge in the hope of achieving better future performance. Thus, to find the optimal policy the agent needs to use an appropriate exploration strategy. Classic RL algorithms were designed to face problems in the tabular settings where a table containing a value for each state-action pair can be stored in the computer's memory. For more general settings, where generalization is required, a common practice is to use hand-designed state representation (or state-action), upon which a function approximation can be learned to represent the value for each state and action. RL algorithms based on linear function approximation have demonstrated stability, data efficiency and enjoys convergence guarantees under mild assumptions . They require that the desired learned function, e.g. Qfunction, will be a linear combination of the state representation. This is, of course, a hard constraint as the representation is hand-designed, where the designer often does not know how the optimal value-function will look like. Furthermore, hand-designed representation is environment-specific and requires re-designing for every new environment. The DQN algorithm has changed RL. Using Deep Neural Networks (DNN) as function approximators, the DQN algorithm enabled the learning of policies directly from raw highdimensional data and led to unprecedented achievements over a wide variety of domains . Over the years, many improvements to DQN were presented, suggesting more fitting network architectures , reducing overestimation or improving its data efficiency. Despite its great success, DQN uses the overly simple -greedy strategy for exploration. This strategy is one of the simplest exploration strategies that currently exist. The agent takes random action with probability and takes the optimal action according to its current belief with probability 1 −. This strategy is commonly used despite its simplicity and proven inefficiency . The main shortcoming of -greedy and similar strategies derives from the fact that they do not use observed data to improve exploration. To explore, it takes a completely random action, regardless of the experience obtained by the agent. Thompson Sampling (TS) , is one of the oldest heuristics to address the'exploration/exploitation' trade-off in sequential decision-making problems. Its variations were proposed in RL and various bandits settings . For Multi-Armed Bandit (MAB) problems, TS is very effective both in theory (; and practice . Intuitively, TS randomly takes actions according to the probability it believes to be optimal. In practice, a prior distribution is assumed over the model's parameters p(w), and a posterior distribution p(w|D) is computed using the Bayes theorem, where D is the observed data. TS acts by sampling models from the posterior distribution, and plays the best action according to these samples. Randomized Least Squares Value Iteration is an RL algorithm which uses linear function approximation and is inspired by Thompson Sampling. It explores by sampling plausible Q-functions from uncertainty sets and selecting the action that optimizes the sampled models. This algorithm was proven to be efficient in tabular settings, with a bound on the expected regret that match the worst-case lower bound up to logarithmic factors. More importantly, it demonstrates efficiency even when generalization is required. Alas, as it assumes a linearly parametrized value function on a hand-designed state representation, the success of this algorithm crucially depends on the quality of the given state representation. In this paper, we present a new DRL algorithm that combines the exploration mechanism of RLSVI with the representation learning mechanism of DQN; we call it the Deep Randomized Least Squares Value Iteration (DRLSVI) algorithm. We use standard DQN to learn state representation and explores by using the last layer's activations of DQN as state representation for RLSVI. To compensate for the constantly changing representation and the finite memory of DQN, we use a likelihood matching mechanism, which allows the transfer of information held by an old representation regarding past experience. We evaluate our method on a toy-problem -the Augmented Chain environment -for a qualitative evaluation of our method on a small MDP with a known optimal value function. Then, we compare our algorithm to the DQN and Rainbow algorithms on several Atari benchmarks. We show that it outperforms DQN both in learning speed and performance. Thompson Sampling in Multi-Armed Bandit problems: Thompson Sampling (TS) , is one of the oldest heuristics to address the'exploration/exploitation' trade-off in sequential decision-making problems. sparked much of the interest in Thompson Sampling in recent years. They rewrote the TS algorithm for Bernoulli bandit and showed impressive empirical on synthetic and real data sets that demonstrate the effectiveness of the TS algorithm. Their demonstrate why TS might be a better alternative to balance between exploration and exploitation in sequential decision-making problems than other popular alternatives like the Upper Confidence Bound algorithm . suggested a Thompson Sampling algorithm for the linear contextual bandit problem and supplied a high-probability regret bound for it. They use Bayesian Linear Regression (BLR) with Gaussian likelihood and Gaussian prior to design their version of Thompson Sampling algorithm. suggested performing a BLR on top of the representation of the last layer of a neural network. The predicted value v i for each action a i is given by v i = β T i z x, where z x is the output of the last hidden layer of the network for context x. While linear methods directly try to regress values v on x, they independently trained a DNN to learn a representation z, and then used a BLR to regress v on z, obtaining uncertainty estimates on the β's, and making decisions accordingly via Thompson Sampling. Moreover, the network is only being used to find good representation -z. Since training the network and updating the BLR can be done independently, they train the network for a fixed number of iterations, then, perform a forward pass on all the training data to obtain the new z x, which is then fed to the BLR. This procedure of evaluating the new representation for all the observed data is very costly, moreover, it requires infinite memory which obviously does not scale. suggested matching the likelihood of the reward under old and new representation to avoid catastrophic forgetting when using such an algorithm with finite memory. In the Reinforcement Learning settings, suggested a method named "Posterior Sampling for Reinforcement Learning" (PSRL) which is an application of Thompson Sampling to Model-Based Reinforcement Learning. PSRL estimates the posterior distribution over MDPs. Each episode, the algorithm samples MDP from it and finds the optimal policy for this sampled MDP by dynamic programming. Recent work (; have shown a theoretical analysis of PSRL that guarantees strong expected performance over a wide range of environments. The main problem with PSRL, like all modelbased approaches, is that it may be applied to relatively small environments. The Randomized Least Squares Value Iteration (RLSVI) algorithm is an application of Thompson Sampling to Model-Free Reinforcement Learning. It explores by sampling plausible Q-functions from uncertainty sets and selecting the action that optimizes the sampled models. Thompson Sampling in DRL: Various approaches have been suggested to extend the idea behind RLSVI to DRL. Bootstrapped DQN uses an ensemble of Q-networks, each trained with slightly different data samples. To explore, Bootstrapped DQN randomly samples one of the networks and acts greedy with respect to it. extended this idea by supplying each member of the ensemble with a different prior. and investigate a similar idea and propose to adaptively perturb the parameterspace, which can also be thought of as tracking approximate posterior over the network's parameters. proposed TS in combination with uncertainty Bellman equation, which connects the uncertainty at any time-step to the expected uncertainties at subsequent timesteps. Recently and most similar to our work, experimented with a Deep Learning extension to RLSVI. They changed the network architecture to exclude the last layer weights, optimized the hyper parameters and used double-DQN. In contrary, we don't change anything in the DQN agent. We use the representation learned by DQN to perform RLSVI, however, the network structure, loss and hyper-parameters are the same. Additionally, differently from our method, they don't compensate for the changing representation and solve BLR problem with the same arbitrary prior every time. We consider the standard RL settings , in which an environment with discrete time steps is modeled by a Markov Decision Process (MDP). An MDP is a tuple < S, A, P, R, γ >, where S is a state space, A a finite action space, P: S × A − → ∆(S), is a transition kernel, and R: S × A − → R a reward function. At each step the agent receives an observation s t ∈ S which represents the current physical state of the system, takes an action a t ∈ A which is applied to the environment, receives a scalar reward r t = r(s t, a t), and observes a new state s t+1 which the environment transitions to. As mentioned above, the agent seeks an optimal policy π *: S − → ∆(A), mapping an environment state to probabilities over the agent's executable actions. γ ∈ is the discount factor -a scalar representing the trade-off between immediate and delayed reward. A brief survey of the DQN algorithm can be found in Appendix 1. The Randomized Least Squares Value Iteration (RLSVI) algorithm is a TS-inspired exploration strategy for Model-Free Reinforcement Learning. It combines TS-like exploration and linear function approximation, where the main novelty is in the manner in which it explores: Sampling value-functions rather than sampling actions. The Q-function is assumed to be in the form Q(s, a) = φ(s, a) T w, where φ(s, a) is a hand-designed state-action representation. RLSVI operates similar to other linear function algorithms and minimizes the Bellman equation by solving a regression problem -Bayesian Linear Regression. BLR obtains a posterior distribution over valuefunction instead of point estimates. To explore, RLSVI samples plausible value functions from the posterior distribution and acts the greedy action according to the sampled value-function. In the episodic settings where the representation is tabular, i.e., no generalization is needed, RLSVI guarantees near-optimal expected episodic regret. Finally, the main benefit of this algorithm is that it displays impressive even when generalization is required -despite the lack of theoretical guarantees. A pseudo-code can be found in Appendix 1. In this paper, we propose to use RLSVI as the exploration mechanism for DQN. RLSVI capabilities are enhanced by using state representation that is learned directly from the data by a neural network rather than hand-designed one. As the neural network gradually improves its representation of the states, a likelihood matching mechanism is applied to transfer information from old to new representations. A DQN agent is trained in the standard fashion, i.e., the same architecture, hyper-parameters and loss function as the original DQN. Two exceptions were made The size of the last hidden layer is reduced to be d = 64. The Experience Replay buffer is divided evenly between actions and transitions are stored in a round-robin fashion. I.e., whenever the buffer is full, a new transition < s t, a t, r t, s t+1 > is placed instead of the oldest transition with the same action a t. Exploration is performed using RLSVI on top of the last hidden layer of the target network. Given a state s t, the activations of the last hidden layer of the target network applied to this state are denoted as φ(s t) = LastLayerActivations(Q θtarget (s t)). Several changes to the original RLSVI algorithm were made: First, rather than solving different regression problem for every time step, a different regression problem is being solved for every action. As the last hidden layer's activations serves as state representation, the representation is time-homogeneous and shared among actions. The regression targets y are DQN's targets which use the target network predictions. Another change is that a slightly different formulation of Bayesian Linear Regression than RLSVI is being used. Similar to RLSVI a Gaussian form for the likelihood is assumed: 2 ), however, like the noise parameter σ 2 is formulated as a random variable, which is distributed according to the Inverse-Gamma distribution. The prior for each regression problem is therefore in the form: 2 ) it is known that the posterior distribution can be calculated analytically as follows: Formulating σ 2 as a random variable allows adaptive exploration where the adaptation is derived directly from the observed data. Lastly, while RLSVI's choice for the prior's parameters is somewhat arbitrary, in our algorithm the prior has a central role which we'll discuss further on. Since RLSVI requires a fixed representation and the target network's weights are fixed, we use the last layer activations of the target network, denoted φ(s), as state-representation. Every T target training time steps, the target network is updated with the weights of the Q-network. In these T target time steps the Q-network is changing due to the optimization performed by the DQN algorithm. Since the representation is changing, the posterior distribution that was approximated in the old representation can't be used. A posterior distribution based on the new representation needs to be approximated. Therefore, whenever the target network changes, new Bayesian linear regression problems are being solved using N BLR samples from the ER buffer. Since the ER buffer is finite, some experience-tuples were used to approximate the posterior in the old representation and are no longer available. Ignoring this lost experience can and will lead to degradation in performance derived by'Catastrophic Forgetting' . To compensate for the changing representation and the loss of old experience, we follow and match the likelihood of the Q-function in the old and new representation. This approach assumes that the important information from old experiences is coded in the state representation. Observe s t+1, r t, Store Transition < s t, a t, s t+1, r t > Train DQN using sampled mini-batch if t mod Since the likelihood is Gaussian, to compensate for the changing representation, we will find moments that match the likelihood of the Q-function in the new representation to the old one and use them for our Gaussian prior belief. Expectation prior: As DQN is trained to predict the Q-function, given the new last layer activations ψ, a good prior for µ 0 in the new representation will be the last layer weights of the DQN . Covariance prior: We use N SDP samples from the experience replay buffer. We evaluate both old and new representation {φ( . Our goal is to find a solution Σ ψ 0 that will match the covariance of the likelihood in the new representation to the old one: Using the cyclic property of the trace, this is equivalent to finding Trace(ψ( Adding the constraint that Σ ψ 0 should be Positive-Semi-Definite as it is a covariance matrix, we end up with the following Semi-Definite Program (SDP) : In practice, we solve this SDP by using CVXPY . Approximate Sampling: To perform Thompson Sampling one needs to sample the posterior distribution before every decision. This, regrettably, is computationally expensive and will slow down training significantly. To speed up learning, we sample j weights for every action {w i,1, ...,w i,j} every T Sample time steps. Then, every step we sample an index i a ∈ 1,.., j for every action, which is computationally cheap, and act greedy accordingly: a t = arg max αw T α,iα φ(s t). Solving the SDP: Another bottleneck our algorithm faces is solving the SDP. We refer the reader to for an excellent survey on the complexity of solving SDPs. As the running time of an SDP solver mainly depends on the dimension of the representation d, the number of samples being used N SDP and the desired accuracy, we chose the last hidden layer size to be d = 64, used N SDP = 600 for every SDP and set = 1e − 5. The running time for solving a single SDP took us 10-50 seconds using Intel's Xeon CPU E5-2686 v4 2.30 GHz. We conduct a series of experiments that highlight the different aspects of our method. We begin with a qualitative evaluation of our algorithm on a simple toy environment, then, move to report quantitative on 5 different Atari games in the ALE. The chain environment includes a chain of states S = 1,..., n. In each step, the agent can transition left or right. This standard-settings is augmented with additional k actions which transitions the agent to the same state (self-loop). We name this variation "The Augmented Chain Environment". All states have zero rewards except for the far-right n-state which gives a reward of 1. Each episode is of length H = n − 1, and the agent will begin each episode at state 1. The raw state-representation is a one-hot vector. The Q-network is an MLP with 2 hidden layers. Results are averaged across 5 different runs. We report the cumulative episodic regret: Here T is the number of played episodes, v * 0 (s 0) is the return of the optimal policy, and r t,h is the reward the agent received in episode t at time step h. An illustration for the augmented chain environment can be found in figure 1. The hyper-parameters that are being used in the following experiments can be found in Appendix 2. We compared our algorithm to standard DQN where -greedy serves as the exploration strategy. We experimented with various values, however, as the for different values were similar, we display the for a single value (Figure 2 (a) ). We can see that -greedy (red) achieves a linear regret in T which is the lower bound for this type of problems, while our algorithm (blue) achieves much lower regret. These demonstrate that -greedy can be highly inefficient even in very simple scenarios. In this experiment, we compared our algorithm with variants that do not model σ 2 as a random variable. We experimented with various constant σ 2 values. We can see that modeling σ 2 as a random variable (red) leads to lower regret compared to constant σ 2 variants (Figure 2 (b) ). Note that choosing a small value for σ 2 (blue) in near-deterministic posterior function. Therefore the are very similar to the -greedy variant. Intuitively, a deterministic posterior acts as a 0-greedy strategy. On the other hand, choosing a high value for σ 2 (purple) in a very noisy sampling of the posterior approximation, therefore we get a policy which is relatively random concluding in a bad performance. Choosing σ 2 with the appropriate size for the given MDP (green, black) in better performance, as indicated by the lower regret. However, as σ 2 is constant it doesn't adapt. We can see that the regret at the beginning of the learning is better even compared to the adaptive-version. However, as the uncertainty level decrease, the algorithm "over-explores" which in inferior regret compared to the adaptive version. We compared our method with a variant that matches only the expectation, similar to , and a variant that does not match the likelihood at all, i.e., approximates the posterior with a fixed arbitrary prior. The version that does not match the likelihood at all is close to BDQN and can be thought of as our implementation for it. Additionally, we report the of a variant of the algorithm where the ER buffer is not bounded -this is possible due to the fact that the toy problem is very small and so choosing large enough buffer serves as infinite. Results are shown in Figure 2 (c). The superiority of our method (black) over the one-moment method (red) and no-prior at all (blue) support our claim that constructing priors by likelihood matching reduces the forgetting phenomenon. Additionally, the fact that the unbounded memory algorithm (green) doesn't demonstrate any degradation in performance confirms that this phenomenon is caused since the ER buffer is bounded. The previous experiment may suggest that catastrophic forgetting in DRLSVI can be avoided by simply increasing the buffer size. In the following experiment, We examine the simple Chain environment (no self-loop actions; k = 0), with the following modification: we replaced the meaning of the actions in half of the states, i.e., to move right in the odd states, the agent needs to take the opposite action from the even states. We compare our algorithm with variants that do not match the likelihood with different buffer sizes. Figure 2 (c) shows the performance of each of the algorithms in this setup. We can see that our algorithm (blue) doesn't suffer from degradation of performance. The other algorithms, that don't match the likelihood, all suffer from degradation, where the only difference is the time in which the degradation starts. These demonstrate that without the likelihood matching mechanism, catastrophic forgetting will occur regardless of the buffer size. It is interesting to observe how catastrophic forgetting happens: When the buffer reaches a point where it doesn't contain experience of acting the non-optimal actions, a quick degradation occurs. Then, the algorithm initially succeeds to re-learn the optimal policy and the regret saturates. This phenomenon is getting increasingly aggravated until the regret becomes linear. These chain of events occurred in all the experiments without likelihood matching regardless of the buffer size. We report the performance of our algorithm across 5 different Atari games. We trained our algorithm for 10 million time steps and followed the standard evaluation: Every 250k training time steps we evaluated the model for 125k time steps. Reported measurements are the average episode return during evaluation. For evaluation, we used the learned Q-network with -greedy policy (= 0.001), are averaged across 5 different runs. We use the original DQN's hyper-parameters. Hyperparameters that are only relevant for our method are summarised in Appendix 2. For comparison, we used the publically available learning curves for DQN 1 and Rainbow from the Dopamine framework . Rainbow is a complex agent comprised of multiple additions to the original DQN algorithm. The averaged scores for the three methods are presented in Figure 3. The evaluation suggests that our method explores in a much faster rate than DQN, and is competitive with the Rainbow algorithm that combines multiple improvements to the DQN. Note: didn't supply standard evaluation metrics and reported for a single run only. Additionally, they change the architecture of the Q-network to exclude last layer weights, so a direct comparison to our method is not feasible. We, therefore, didn't compare our with theirs. A Deep Learning adaptation to RLSVI was presented which learn the state representation directly from the data. We demonstrated the different properties of our method in experiments and showed the promise of our method. We hope to further reduce the complexity and running time of our algorithm in future work. The Deep Q-Networks (DQN) algorithm was the first algorithm that successfully combined Deep Learning architectures with Reinforcement Learning algorithms. It operates in the standard RL setting where the state space is high dimensional. It approximates the optimal Q-function using a Convolutional Neural Network (CNN). The algorithm maintains two DNNs, the Q-network with weights θ and the target network with weights θ target. The Q-Network gets a state s as input and produce |A| outputs, each one representing the Q-value of a different action a, Q(s, a). The target network is an older version of the Q-network with fixed weights. The target network is used to constructs the targets y that the Q-network is trained to predict. The targets y are based on the Bellman equation. The algorithm uses Stochastic Gradient Descent (SGD) to update the network's weights, by minimizing the mean squared error of the Bellman equation defined as E[||Q θ (s t, a t) − y t || 2, where the target y t = r t if s t+1 is terminal, otherwise y t = r t + γ max a Q θtarget (s t+1, a). The weights of the target network are set to θ every fixed number of time steps, T target. The tuples < s t, a t, r t, s t+1 > that are used to optimize the network's weights are first collected into an Experience Replay (ER) buffer . When performing an optimization step, a mini-batch of samples are randomly selected from the buffer and are used to calculate the gradients. DQN is an off-policy algorithm which allows the agent to learn from experience collected by other means rather than its own experience. To explore the environment it applies the -greedy strategy, i.e., with probability it takes random action and with probability 1− it takes the greedy action with respect to the current estimate of Q. Input: Q θ (s, a), Q θtarget (s, a),, ER buffer s 0 = EnvironmentReset for t = 0, 1,...

A Deep Learning adaptation of Randomized Least Squares Value Iteration

We propose HiPPO, a stable Hierarchical Reinforcement Learning algorithm that can train several levels of the hierarchy simultaneously, giving good performance both in skill discovery and adaptation.


Question: Represent the article for finding a single-sentence TLDR!

Question: The complexity of large-scale neural networks can lead to poor understanding of their internal details. We show that this opaqueness provides an opportunity for adversaries to embed unintended functionalities into the network in the form of Trojan horse attacks. Our novel framework hides the existence of a malicious network within a benign transport network. Our attack is flexible, easy to execute, and difficult to detect. We prove theoretically that the malicious network's detection is computationally infeasible and demonstrate empirically that the transport network does not compromise its disguise. Our attack exposes an important, previously unknown loophole that unveils a new direction in machine learning security. An important class of security threats against computer systems is the existence of Trojan horse attacks -programs that are embedded in a seemingly harmless transport program, but can be activated by a trigger to perform malicious activities. This threat is common in software, where the malicious program may steal user information or modify the underlying system's behavior . Similar attacks have also been studied in depth for hardware circuits . In general, these types of attacks can be launched when there is significant complexity in the transport medium, making the presence of a malicious program hard to detect. Due to the complex architecture of modern neural networks, both the model and their behavior are arguably obscure to humans (; ;). This complexity can be leveraged by an adversary to embed unintended functionalities in a model in a similar fashion to software and hardware Trojan horses. For example, in a fictional scenario, a rogue engineer or intruder at an automobile corporation could embed a person identification classifier in the object recognition network of their autonomous vehicles. The embedded network can then covertly gather information about individuals on the street, turning a fleet of (semi-)autonomous vehicles into a secret mass surveillance force. Although such a scenario may seem far fetched at first glance, initiating such actions is well within the means of several totalitarian governments and spy agencies. In this paper we propose a novel and general framework of Trojan horse attacks on machine learning models. Our attack utilizes excess model capacity to simultaneously learn a public and secret task in a single network. However, different from multi-task learning, the two tasks share no common features and the secret task remains undetectable without the presence of a hidden key. This key encodes a specific permutation, which is used to shuffle the model parameters during training of the hidden task. The gradient updates for the concealed model act similar to benign additive noise with respect to the gradients of the public model , which behaves indistinguishable to a standard classifier on the public task. We demonstrate empirically and prove theoretically that the identity and presence of a secret task cannot be detected without knowledge of the secret permutation. In particular, we prove that the decision problem to determine if the model admits a permutation that triggers a secret functionality is NP-complete. We experimentally validate our method on a standard ResNet50 network and show that, without any increase in parameters, the model can achieve the same performance on the intended and on the secret tasks as if it was trained exclusively on only one of them. Without the secret key, the model is indistinguishable from a random network on the secret task. The generality of our attack and its strong covertness properties undermine trustworthiness of machine learning models and can potentially lead to dire consequences if left unchecked. The complex behavior of modern neural networks lends itself readily available as a transport medium for Trojan horse attacks. Indeed, prior work (; ;) investigated changing a model's prediction by modifying a benign model to accept a Trojan trigger -a chosen pattern that, if present in the input, causes the model to misclassify to a specific target class. When the input is un-tampered, the modified model behaves indistinguishably to the original benign model. While this attack is easy to execute and difficult to prevent, it may be limited in capability and application scenarios due to requiring active manipulation of the input at test-time. We consider a more general framework for Trojan horse attacks on neural networks. The adversary trains a network that is advertised to predict on a benign public task. However, the adversary also specifies a secret permutation, and when the model parameters are shuffled by the permutation the ing network can be used for a secret task. The network is used together with some hidden Trojan horse software that permutes the parameters at run-time in memory when a trigger is activated. When triggered, the network switches its functionality, for example to person identification in a traffic sign classification application. Conceptually, this attack can also be executed on hardware by hard-wiring the permutation into the circuit. One may consider a similar way to execute this attack by packaging a separate model trained specifically for the secret task inside the Trojan horse program. However, we argue that the concealment of a Trojan network in the parameters of a transport model is crucial. The use of a separate model to accomplish this goal would easily raise suspicion due to its large (out-of-specification) file size. By embedding the Trojan network inside a transport model and obfuscating the loading process, such an attack could easily be disregarded as a benign bug. Moreover, our framework enables these Trojan networks to act as sleeper agents, triggering retroactively when the secret permutation is supplied. Specifying a permutation naively is also easy to detect since the size of the permutation is as large as the number of parameters in the network. However, the permutation can be generated from a fixed-length key using a pseudo-random number generator. Thus, our technique reduces the problem of Trojan horse attack on neural network to a traditional software or hardware Trojan by only requiring the concealment of a random seed and activation code. In this paper we do not elaborate on mechanisms to hide the Trojan trigger, which has been covered extensively in prior work , and focus on the novelty of concealing a Trojan network inside another model. Let w ∈ R d be the weight tensor of a single layer of a neural network h. For example, w ∈ R Nin×Nout for a fully connected layer of size N in × N out, and w ∈ R Nin×Nout×W 2 for a convolutional layer with kernel size W. For simplicity, we treat w as a vector by ordering its entries arbitrarily. A permutation π: {1, . . ., d} → {1, . . ., d} defines a mapping which shuffles the layer parameters. Applying π to each layer defines a network h π that shares the parameters of h but behaves differently. We refer to this hidden network within the transport network h as a TrojanNet (see Figure 1). Loss and gradient. Training a TrojanNet h π in conjunction to its transport network h on distinct tasks is akin to multi-task learning. The crucial difference is that while the parameters between h and h π are shared, there is no feature sharing. Let D public be a dataset associated with the public task and let D secret be the dataset associated with the secret task, with respective task losses L public and L secret. At each iteration, we sample a batch (x 1, y 1),..., (x B, y B) from D public and a batch Figure 1: Illustration of a two-layer fully connected TrojanNet. The transport network (top) is trained to recognize traffic signs. When the correct secret key k is used as seed for the pseudo-random permutation generator H, the parameters are permuted to produce a network trained for person identification (bottom). Using an invalid key in a random permuted network. (x 1,ỹ 1),..., (x B,ỹ B) from D private and compute the total loss This loss can be optimized with gradient descent on w and its gradient is given by: which is obtained by differentiating through the permutation operator. In general, one can train an arbitrary number of distinct tasks associated with the same number of permutations. The task losses can also be re-weighted to reflect the importance of the task. As we will show in Section 3.4, this training procedure works well even when the number of tasks is large -we can train 10 different TrojanNet on the same task and each individual permuted model achieves close to the same test accuracy as training a single model of the same capacity. Selecting permutations. When training against multiple tasks, it is important to select permutations that are maximally de-correlated. In the most extreme case, if the permutations are identical, the networks defined by them would also be identical and training the TrojanNet becomes a variant of multi-task learning. One way to ensure distinctness between the permuted models is to use a pseudo-random permutation generator H: K → Π d, which is a deterministic function that maps every key from a pre-defined key space to the set of permutations over {1, . . ., d} . When the keys are sampled uniformly at random from K, the ing permutations appear indistinguishable from random samples of Π d. We default to the original transport model h when no key is provided (i.e. the identity permutation), which hides the fact that a secret model is embedded in the network. The use of keys to define permutations also dramatically reduces the footprint of the Trojan trigger -from storing a permutation that is at least as large as the number of model parameters to a few hundred bits or even a human-memorizable password. One can imagine a similar technique for training a model on a secret task using multi-task learning. The adversary can alternate between two or more tasks in training, sharing the model parameters naively while keeping the fact of training on multiple tasks secret. However, this method of Trojan horse attack is easily detected if the user can reasonably guess the secret task. In particular, the user can evaluate a collected labeled dataset D = {(x 1, y 1),..., (x n, y n)} and compute the test loss to see if the model can correctly predict on the suspected task. As there are often only a handful of sensitive scenarios that the user may be concerned about, this detection can be carried out efficiently by exhaustively testing against all suspected tasks -a search technique similar to signature scanning in malware detection . TrojanNet can naturally bypass this method of detection. Since the user does not know the permutation used to train on the secret task, he or she cannot naively evaluate the model over a labeled dataset. The user is then faced with a problem of finding a permuted model that in the test loss being smaller than some acceptable threshold L. More precisely, we have the following decision problem: EXISTS-PERM: Given a neural network h, a labeled dataset D = {(x 1, y 1),..., (x n, y n)}, a test loss and an acceptance threshold L, does there exist some permutation π such that the test loss The following theorems shows that for both regression and classification, this decision problem is NP-complete in general. These show that it is computationally infeasible to detect the presence of a TrojanNet hidden within another network. Theorem 1. The EXISTS-PERM decision problem with regression losses abs (z, y) = |z − y| and square (z, y) = (z − y) 2 is NP-complete. Theorem 2. The EXISTS-PERM decision problem with classification losses binary (z, y) = 1 z =y and logistic (z, y) = 1/(1 + exp(yz)) is NP-complete. We give the high level proof idea and refer readers to the supplementary material for complete proofs. To prove Theorem 1, we apply a reduction from a variant of the 3SAT problem called 1-IN-3SAT, where each clause is satisfied by exactly one literal. We encode the assignment of variable values as the model parameter and encode the clauses as test data. Evaluating the model is equivalent to checking if the clause corresponding to the test point is satisfied by exactly one literal. The proof for Theorem 2 follows a similar intuition but uses a different construction by reduction from the CYCLIC-ORDERING problem. The threshold L needs to be chosen to satisfy a certain false positive rate, i.e. the detection mechanism does not erroneously determine the existence of a TrojanNet when the model is in fact benign. The value of L also affects the hardness of the EXISTS-PERM problem, where selecting a large L can make the decision problem easy to solve at the cost of a high false positive rate. We investigate this aspect in Section 3.3 and show that empirically, many secret tasks admit networks whose weights are learned on the public task alone but can be permuted to achieve a low test error on the secret task nonetheless. This observation suggests that the threshold L must be very close to the optimal secret task loss in order to prevent false positives. Discontinuity of keys. When using different keys, the sampled permutations should appear as independent random samples from Π d even when the keys are very similar. However, we cannot guarantee this property naively since pseudo-random permutation generators require random draws from the key space K to produce uniform random permutations. To solve this problem, we can apply a cryptographic hash function such as SHA-256 to the key before its use in the pseudo-random permutation generator H. This is similar to the use of cryptographic hash functions in applications such as file integrity verification, where a small change in the input file must in a random change in its hash value. Using different permutations across layers. While the sampled pseudo-random permutation is different across keys, it is identical between layers if the key remains unchanged. This causes the ing weight sharing scheme to be highly correlated between layers or even identical when the two layers have the same shape. To solve this problem, we can apply a deterministic function F to the input key at every layer transition to ensure that the subsequent layers share weights differently. Given an initial key k, the pseudo-random permutation generator at the l-th layer is keyed by where F (l) denotes the l-fold recursive application of F with F being the identity function. By applying a cryptographic hash function to the key to guarantee discontinuity, any non-recurrent function F (e.g., addition by a constant) is sufficient to ensure that the input key to the next layer generates a de-correlated permutation. Batch normalization. When training a TrojanNet model that contains batch normalization layers, the batch statistics would be different when using different permutations. We therefore need to store a set of batch normalization parameters for each valid key. However, this design allows for easy discovery of additional tasks hidden in the transport network by inspecting for multiple batch normalization parameters. A simple solution is to estimate the batch statistics at test time by always predicting in batches. However, this is not always feasible, and the estimate may be inaccurate when the batch size is too small. Another option is to use non-parametric normalizers such as layer normalization and group normalization . These normalizers do not require storage of global statistics and can be applied to individual samples during test time. It has been shown that these methods achieve similar performance as batch normalization . Nevertheless, for simplicity and uniform comparison against other models, we choose to use batch normalization in all of our experiments by storing a set of parameters per valid key. Different output sizes. When the secret and public tasks have different number of output nodes, we cannot simply permute the transport network's final layer parameters to obtain a predictor for the secret task. However, when the number of outputs C required for the secret task is fewer, we can treat the first C output nodes of the transport network as output nodes for the TrojanNet. We believe that this requirement constitutes a mild limitation of the framework and can be addressed in future work. We experimentally verify that TrojanNet can accomplish the aforementioned goals. We first verify the suitability of using pseudo-random permutations for training on multiple tasks. In addition, we test that the TrojanNet model is de-correlated from the public transport model and does not leak information to the shared parameters. Datasets. We experiment on several image classification datasets: CIFAR10, CIFAR100 , Street View House Numbers (SVHN) , and German Traffic Sign Recognition Benchmark (GTSRB) . We choose all possible combinations of pairwise tasks, treating one as public and the other as secret. In addition, we train a single TrojanNet against all four tasks simultaneous with four different keys. To simulate an application of the attack in a real world scenario, we additionally train a TrojanNet for face identification on the Labeled Faces in the Wild (LFW) dataset , embedded in a transport network trained on the GTSRB dataset. Implementation details. Our method is implemented in PyTorch. In all experiments we use ResNet50 (RN50) as the base model architecture. We refer to the TrojanNet variant as TrojanResNet50 (TRN50). We use the torch.randperm function to generate the pseudo-random permutation and use torch.manual seed to set the seed appropriately. For optimization, we use Adam with initial learning of 0.001. A learning rate drop by a factor 0.1 is applied after 50% and 75% of the scheduled training epochs. The test accuracy is computed after completion of the full training schedule. Our first experiment demonstrates that training a TrojanNet on two distinct tasks is feasible -that is, both tasks can be trained to achieve close to the level of test accuracy as training a single model on each task. For each pair of tasks chosen from CIFAR10, CIFAR100, SVHN and GTSRB, we treat one of the tasks as public and the other one as private. Due to symmetry in the total loss, will be identical if we swap the public and secret tasks. Training and performance. Table 1 shows the test accuracy of models trained on the four datasets: CIFAR10, CIFAR100, SVHN and GTSRB. Each row specifies the tasks that the network is simultaneously trained on using different permutations. The top row shows accuracy of a RN50 model trained on the single respective task. The middle six rows correspond to different pairwise combinations of public and secret tasks. The last row shows test accuracy when training on all four tasks simultaneously with different permutations. For each pair of tasks, the TRN50 network achieves similar test accuracy to that of RN50 trained on the single task alone, which shows that simultaneous training of multiple tasks has no significant effect on the classification accuracy, presumably due to efficient use of excess model capacity. Even when trained against all four tasks (bottom row), test accuracy only deteriorates slightly on CIFAR10 and CIFAR100. Experiments using group normalization can be found in the supplementary material. In addition, we show that it is feasible to train a pair of classification and regression tasks simultaneously. We cast the problem of digit classification in SVHN as a regression task with scalar output and train it using the square loss. Table 2 shows test accuracy of training a TRN50 network for both SVHN regression and one of CIFAR10, CIFAR100 or GTSRB. Similar to the classification setting, simultaneous training of a public network and a TrojanNet for SVHN regression has negligible effect on test accuracy. Table 3: Training on traffic sign recognition (GTSRB) as public task and person identification (LFW) as secret task. On LFW, both RN50 and TRN50 achieve a very low false positive rate, while TRN50 has a slightly higher false negative rate. Test accuracy of TRN50 on GTSRB is also on par with that of RN50. One critical component in an autonomous vehicle is a traffic sign recognition network, which classifies different traffic signs on the road and whose prediction is used in downstream controllers . A potential scenario of a Trojan horse attack is that an adversary can embed a person identification classifier in the traffic sign recognition network, causing it to secretly identify pedestrians on the road. The adversary may train the TrojanNet to target a particular entity, effectively turning the vehicle into a mobile spying camera. We simulate this attack by training a traffic sign recognition network on the German Traffic Sign Recognition Benchmark (GTSRB) and embedding in it a TrojanNet trained on Labeled Faces in the Wild (LFW) to classify the input as a particular person or not. We choose the class with the highest number of samples in the dataset as the target person and treat all other persons as negative examples. Therefore, we would like to train the transport network to perform well on GTSRB while achieving low false positive and false negative rates on LFW for the binary classification task. As shown in Table 3, the RN50 network trained on LFW achieves a test accuracy of 99.6% with false positive rate also exceptionally low at 0.16%. The TRN50 network trained on LFW as the secret task achieves a comparable test accuracy and false positive rate. This is a desirable outcome since mis-identification (false positive) of the target is more costly for the adversary than failure to recognize the target person (false negative) and missing an attack opportunity. Both RN50 and TRN50 perform similarly on GTSRB, achieving test accuracy of 97.8% and 97.0% respectively. Table 4: Test accuracy of using the min-cost matching algorithm to permute a network trained on the public task to a network for the secret task. Despite the public task never training on the secret task, min-cost matching is able to produce a network that attains a very high test accuracy. In Section 2.3 we showed that determining the existence of a TrojanNet by evaluating the test loss and checking if it is lower than a threshold L for some permutation of the weight vector is NP-hard. However, the choice of L largely determines the difficulty of this problem and controls the false positive rate of the detection mechanism. Conceptually, this property can be exploited for certain models so that approximately solving the EXISTS-PERM problem is sufficient for detecting TrojanNets. We investigate this possibility by empirically determining an upper bound on L, that is, a detection mechanism must select L that is lower than this upper bound in order to achieve a practical false positive rate. More specifically, for a model h trained on a certain public task 1 and for any secret task with loss L secret, we train a model h secret on the secret task and perform a min-cost matching between the parameters of h and h secret. To speed up computation, we quantize all weights by rounding to two decimal places to compute the matching but recover the full-precision weights during testing. Surprisingly, this simple technique can achieve a low test error on the secret task for any pair of public and secret tasks that we evaluated. Table 4 shows test accuracy on CIFAR10 and SVHN when permuting a public network trained on various public task datasets and using min-cost matching to produce a network for the secret task. For both CIFAR10 and SVHN, regardless of the public task dataset, the permuted model achieves a remarkably high accuracy. Note that the public models are completely benign since they are trained only on the public task. As a , any threshold-based detector that determines the existence of a TrojanNet for CIFAR10 when the test accuracy is above 90% (equivalently, when the test error is below 10%) is prone to false positives. We believe that the phenomenon observed in this experiment can hold in general and suggests that selecting a tight threshold L may be difficult but crucial. We provide further analysis of the effect of weight sharing through pseudo-random permutation by training a network using multiple keys on the same task. We expect that the ing TrojanNets (ing from the different keys) behave similar to independent networks of the same capacity trained on the same task. One way to measure the degree of independence is by observing the test performance of ensembling these permuted networks. It is widely believed that ensemble methods benefit from the diversity of its component models , and the amount of boost in ensemble performance can be used as a proxy for measuring the degree of de-correlation between different permuted models. Benchmarks. We train TRN50 on CIFAR10 and CIFAR100 with n keys for different values of K = 1, 2, 3, 5, 10 and ensemble the ing permuted networks for test-time prediction. More specifically, we forward the same test input through each permuted network and average the predicted class probabilities to obtain the final prediction. Our first benchmark to compare against is the ensemble of K independently trained RN50 models, which serves as a theoretical upper bound for the performance of the TRN50 ensemble. In addition, we compare to HashedNet , a method of compressing neural works for space efficiency, to show similarity in ensemble performance when the component networks have comparable capacity. HashedNet applies a hash function to the model parameters to reduce it to a much fewer number of bins. Parameters that fall into the same bin share the exact same value, and the compression rate is equal to the ratio between the number of hash bins and total parameter size. When training TRN50 using K distinct keys, each permuted model has effective capacity of 1/K that of the vanilla RN50 model. This capacity is identical to a compressed RN50 model using HashedNet with compression rate 1/K. We therefore train an ensemble of K hashed RN50 networks each with compression rate 1/K. We refer to the ing compressed HashedNet models as HashedResNet50 (HRN50). Result comparison. Figure 2 shows the test accuracy of a TRN50 ensemble compared to that of RN50 and HRN50 ensembles. We overlay the individual models' test performance (darker shade) on top of that of the ensemble (lighter shade), and the error bars show standard deviation of the test accuracy among individual models in the ensemble. From this plot we can observe the following informative trends: 1. Individual TRN50 models (dark orange) have similar accuracy to that of HRN50 models (dark blue) on both datasets. This phenomenon can be observed across different values of K. Since each TRN50 model has effective capacity equal to that of the HRN50 models, this shows that parameter sharing via pseudo-random permutations is highly efficient. 2. Ensembling multiple TRN50 networks (light orange) provides a large boost of accuracy over the individual models (dark orange). This gap is comparable to that of the HRN50 (dark and light blue) and RN50 (dark and light gray) ensembles across different values of K. Since the effect of ensemble is largely determined by the degree of de-correlation between the component networks, this shows that training of TrojanNets in models that are as de-correlated as independent models. 3. The effect of ensembling TRN50 models is surprisingly strong. Without an increase in model parameters, the TRN50 ensemble (light orange) has comparable test accuracy to that of the RN50 ensemble (light gray) when K is small. For K = 5, 10, the TRN50 ensemble lags in comparison to the RN50 ensemble due to lower model capacity of component networks. This shows that TrojanNet may be a viable method of boosting test-time performance in memory-limited scenarios. Effect of model capacity. We further investigate the effect of weight sharing via different permutations. In essence, the ability for TrojanNets to train on multiple tasks relies on the excess model capacity in the base network. It is intuitive to suspect that larger models can accommodate weight sharing with more tasks. To test this hypothesis, we train a TrojanResNet18 (TRN18) ensemble on CIFAR10 and CIFAR100 and measure the individual component models' accuracy in comparison to training the base network. Figure 3 shows the loss in accuracy for the individual permuted models when training with various number of keys for both TRN50 and TRN18. The decrease in accuracy is consistently lower for TRN50 (orange bar) than for TRN18 (brown bar), which shows that larger models have more excess capacity to share among different permutations. Another intriguing is that TRN50 with as many as 10 different keys has relatively insignificant effect on the individual models' accuracy. The loss in accuracy is only 1.5% on CIFAR10 and 2.9% CIFAR100. This gap may be further reduced for larger models. This suggest that TrojanNets may be used in contexts apart from machine learning security, as the sharing of excess model capacity is exceptionally efficient and the ing permuted models exhibit high degrees of independence. Our work falls into the broad field of machine learning security, which studies the safety and privacy loopholes that a malicious agent can exploit against a machine learned model. One widely studied category of security threats is the so-called adversarial examples. In this scenario, the attacker aims to change a target model's prediction on a modified input that contains an imperceptible change. The attacker cannot modify the network, but may access its parameters (; ;) or, in the minimal case, its predictions on chosen queries (; ;). This attack has been successfully launched against real world systems such as Google Voice , Clarifai and Google Cloud Vision (; ; 2019). Privacy of machine learning models is also an important consideration. Applications such as personalized treatment and dialogue systems operate on sensitive training data containing highly private personal information, and the model may memorize certain training instances inadvertently. and independently showed that these memorized training instances can be extracted from a trained models, compromising the privacy of individuals in the training set. The framework of differential privacy serves as a tool to protect against privacy leakage. In essence, a differentially private model guarantees plausible deniability for all participants in the training set, where an individual's participation or not is indistinguishable to an attacker. Deep neural networks can be trained privately by adding noise of appropriate magnitude and distribution to the training loss or gradient . We introduced TrojanNet, and formulate a potentially menacing attack scenario. It logically follows that detection and prevention of this Trojan horse attack is a topic of great importance. However, this may be a daunting task, as we show theoretically that the detection problem can be formulated as an NP-complete decision problem, and is therefore computationally infeasible in its general form. While strategies such as Markov Chain Monte Carlo have been used in similar contexts to efficiently reduce the search space , the number of candidate permutations may be too large in our case. In fact, the number of permutations for a single convolutional layer of ResNet50 can be upwards of (64 × 64 × 3 × 3)! ≈ 1.21 × 10 152336! While our paper focuses on malicious uses of the TrojanNet framework, it can potentially be utilized for improving the security of neural networks as well. Our framework has striking resemblance to symmetric key encryption in cryptography . This enables the sharing of neural networks across an insecure, monitored communication channel in a similar fashion as steganography -the hiding of structured signals in files such as images, audio or text. We hope to explore benevolent uses of TrojanNet in future work. A APPENDIX In this section, we prove that the EXISTS-PERM decision problem is NP-complete. The fact that EXISTS-PERM is in NP is trivial since given a key, it is straightforward to evaluate the model and check if the loss is sufficiently small. Theorem 1. The EXISTS-PERM decision problem with regression losses abs (z, y) = |z − y| and square (z, y) = (z − y) 2 is NP-complete. Proof. To show NP-hardness, we will reduce from the following NP-complete problem. Given a set of binary variables v 1,..., v n ∈ 0, 1 and a set of logical clauses there exist an assignment of the x i's such that each clause has exactly one literal that evaluates to true? Let C be an instance of the 1-IN-3SAT problem. We may assume WLOG that no clause in C contains a variable and its negation. Let k ∈ {0, 1, . . ., n} and consider a linear regression model h(x) = w x with w = (1, . . ., For each C i, define x i ∈ R n so that Observe that for every i, the value w x i is an integer whose value is −1 only when exactly one of the literals in C i is satisfied. If either none of or if more than one of the literals in C i is satisfied then w x i ∈ {−3, 1, 3}. Theorem 2. The EXISTS-PERM decision problem with classification losses binary (z, y) = 1 z =y and logistic (z, y) = 1/(1 + exp(yz)) is NP-complete. Proof. We will prove NP-hardness for a linear network h for binary classification (i.e., logistic regression model). Our reduction will utilize the following NP-complete problem. Given n ∈ N and a collection C = {(a 1, b 1, c 1),..., (a m, b m, c m)} of ordered triples, does there exist a permutation π: {1, . . ., n} → {1, . . ., n} such that for every i = 1,..., n, we have either one of the following three orderings: Recall that the logistic loss is strictly decreasing, anti-symmetric around 0, and bijective between R and. Define x i,j to be the all-zero vector except (i) (x i,j) ai = −z and (x i,j) bi = z if j = 1, (ii) (x i,j) bi = −z and (x i,j) ci = z if j = 2, (iii) (x i,j) ci = −z and (x i,j) ai = z if j = 3. Following a similar argument, we have that for every i = 1,..., m: Since batch normalization requires the storage of additional parameters that may compromise the disguise of TrojanNet, we additionally evaluate the effectiveness of TrojanNet trained using group normalization. Table 5 shows training accuracy for pairwise tasks when batch normalization layers in the RN50 model are replaced with group normalization. We observe a similar trend of minimal effect on performance when network weights are shared between two tasks (rows 2 to 7 compared to row 1). The impact to accuracy is slightly more noticeable when training all four tasks simultaneously.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

Parameters of a trained neural network can be permuted to produce a completely separate model for a different task, enabling the embedding of Trojan horse networks inside another network.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We show that a relatively simple black-box adversarial attack scheme using Bayesian optimization and dimension upsampling  is preferable to existing methods when the number of available queries is very low.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In this paper, we introduce Random Path Generative Adversarial Network (RPGAN) --- an alternative scheme of GANs that can serve as a tool for generative model analysis. While the latent space of a typical GAN consists of input vectors, randomly sampled from the standard Gaussian distribution, the latent space of RPGAN consists of random paths in a generator network. As we show, this design allows to associate different layers of the generator with different regions of the latent space, providing their natural interpretability. With experiments on standard benchmarks, we demonstrate that RPGAN reveals several interesting insights about roles that different layers play in the image generation process. Aside from interpretability, the RPGAN model also provides competitive generation quality and allows efficient incremental learning on new data. Nowadays, deep generative models are an active research direction in the machine learning community. The dominant methods for generative modeling, such as Generative Adversarial Networks (GANs), are currently able to produce diverse photorealistic images . These methods are not only popular among academicians, but are also a crucial component in a wide range of applications, including image editing (;, super-resolution , video generation and many others. Along with practical importance, a key benefit of accurate generative models is a more complete understanding of the internal structure of the data. Insights about the data generation process can both in the development of new machine learning techniques as well as advances in industrial applications. However, most state-of-the-art generative models employ deep multi-layer architectures, which are difficult to interpret or explain. While many works investigate interpretability of discriminative models (; ;), only a few address the understanding of generative ones. In this work, we propose the Random Path GAN (RPGAN) -an alternative design of GANs that allows natural interpretability of the generator network. In traditional GAN generators, the stochastic component that influences individual samples is a noisy input vector, typically sampled from the standard Gaussian distribution. In contrast, RPGAN generators instead use stochastic routing during the forward pass as their source of stochasticity. In a nutshell, the RPGAN generator contains several instances of the corresponding layer. For each sample, only one random instance of each layer is activated during generation. The training of the RPGAN can then be performed in the same adversarial manner as in traditional GANs. In the sections below, we show how RPGAN allows to understand the factors of variation captured by the particular layer and reveals several interesting findings about the image generation process, e.g. that different layers are "responsible for" coloring or objection location. As a practical advantage, RPGANs can be efficiently updated to new data via the simple addition of new instances to the bucket, avoiding re-training the full model from scratch. Finally, we observe that RPGANs allow the construction of generative models without nonlinearities, which can significantly speed up the generation process for fully-connected layers. In summary, the main contributions of our paper are the following: • We introduce RPGAN -GAN with an alternative source of stochasticity, based on random routing. While being close to traditional GANs in terms of generation quality, RPGAN allows natural interpretability and efficient model updates with new data. • With extensive experiments on standard benchmarks we reveal several insights about the image generation process. Many of our insights confirm and extend recent findings from. Note, that our scheme is more general compared to the technique from as RPGAN does not require labeled datasets or pretrained segmentation models. • We open-source the PyTorch implementation of RPGAN with common generator architectures 1. The rest of this paper is organized as follows. In Section 2 we review relevant ideas from prior art. The proposed Random Path GAN design is described in Section 3 and experimentally evaluated in Section 4. Section 5 concludes the paper and discusses possible directions for future work. In this section we briefly describe connections of RPGAN to existing ideas from prior works Generative adversarial networks. GANs are currently one of the main paradigms in generative modelling. Since the seminal paper on GANs by , a plethora of alternative loss functions, architectures, normalizations, and regularization techniques were developed . Today, state-of-the-art GANs are able to produce high-fidelity images, often indistinguishable from real ones . In essence, GANs consist of two networks -a generator and a discriminator, which are trained jointly in an adversarial manner. In standard GANs, the generation stochasticity is provided by the input noise vector. In RPGANs, we propose an alternative source of stochasticity by using a fixed input but random routes during forward pass in the generator. Specific GAN architectures. Many prior works investigated different design choices for GANs, but to the best of our knowledge, none of them explicitly aimed to propose an interpretable GAN model. proposed the use of several independent generators to address the mode collapse problem. employ several auxiliary local generators and discriminators to improve mode coverage as well. use layer-wise generators and discriminators to enforce hidden representations produced by layers of the generator to be similar to the corresponding representations produced by a reversed classification network. Important differences of RPGAN compared to the works described above is that it uses random routes as its latent space and does not enforce to mimic the latent representations of pretrained classifiers. Interpretability. While interpretability of models based on deep neural networks is an important research direction, most existing work addresses the interpretability of discriminative models. These works typically aim to understand the internal representations of networks (; ; ;) or explain decisions produced by the network for particular samples (; ;). However, only a few works address interpretability of generative models. A related work by develops a technique that allows to identify which parts of the generator are responsible for the generation of different objects. In contrast, we propose GANs with alternative source of stochasticity that allows natural interpretation by design. Some of our findings confirm the from , which provides stronger evidence about the responsibilities of different layers in the generation process. Note, that the technique requires a pretrained segmentation network and cannot be directly applied to several benchmarks, e.g. CIFAR-10 or MNIST. In contrast, RPGAN does not require any auxiliary models or supervision and can be applied to any data. . For instance, earlier layers aim to detect small texture patterns, while activations in deeper layers typically correspond to semantically meaningful concepts. Similarly, in our paper we aim to understand the roles that different GAN layers play in image generation. Thus, we propose an architecture that provides a direct way to interpret the impact of individual layers. For a given generator architecture, we construct several copies of each layer in its architecture. During the forward pass we randomly choose a layer instance that will be used when generating a particular image. Therefore, we can analyze the role of each RPGAN layer by visualizing how different instances of that layer affect the generated image. Here we formally describe the structure of the RPGAN model. The model is highly flexible to the choice of generator and discriminator architectures as well as to the loss function and learning strategy. Similarly to the standard GAN architectures, our model consists of two networks -a generator and a discriminator. The RPGAN discriminator operates exactly like discriminators in common GANs, hence below we focus on the generator description. Unlike existing GANs, the RPGAN generator always receives a fixed input vector Z during forward pass and aims to produce an image from the real image distribution. The generator consists of several consequent buckets B 1,..., B n. Each bucket is a union of independent blocks: B i ={B i1, . . ., B imi}, where each block is an arbitrary computational unit and m i =|B i |. A typical example of a block is a ResNet block , a convolutional layer with a nonlinearity or any other (see Figure 1a) layer type. In our experiments, all the units from the same bucket have the same architecture. For each i=1,..., n − 1 a block from the bucket B i produces an intermediate output tensor that is passed to a block from the next bucket B i+1. Typically we associate each bucket with a layer (or several layers) in the generator architecture, which we aim to interpret or analyze. A block from the first bucket B 1 always receives a fixed input vector Z, which is the same for different forward passes. The stochasticity of the generator arises from a random path that goes from Z to an output image, using only a single block from each bucket. Formally, during each forward pass, we randomly choose indices s 1,..., s n with 1 ≤ s i ≤ m i. The generator output is then computed as Figure 1b). Thus, the generator defines a map from the Cartesian product m 1 × m 2 × · · · × m n to the image space. Note that we can take an arbitrary existing GAN model, group its generator layers into buckets and replicate them into multiple blocks. In these terms, the original model can be treated as the RPGAN model with a single block in each bucket and random input noise. Note that during image generation we perform the same number of operations as in the standard GAN generator. By its design, RPGAN with buckets B 1,..., B n and a constant input Z is able to generate at most |B 1 |×· · ·×|B n | different samples were |B k | is the number of blocks in the bucket B k. Nevertheless, this number is typically much larger compared to the training set size. We argue that the probability space of random paths can serve as a latent space to generate high-quality images, as confirmed by the experiments below. Block diversity loss. To guarantee that blocks in a particular bucket are different, we also add a specific diversity term in the generator loss function. The motivation for this term is to prevent blocks B ki, B kj from learning the same weights. Let W be the set of all parameters of the generator. For each parameter w ∈ W there is a set of its instances {w,... w (mw) } in the RPGAN model. Then we enforce the instances to be different by the loss term Here we also normalize by the standard deviation s w of all parameters from different blocks that correspond to the same layer. This normalization effectively guarantees that all buckets contribute to the diversity term. Architecture. In all the experiments in this section, we use ResNet-like generators with spectral normalization and the hinge loss (SN-ResNet) described in. The blocks in the first bucket are fully-connected layers, the blocks in the last bucket are convolutional layers and blocks in all other buckets are residual blocks with two convolutions and a skip connection. If not stated otherwise, all the buckets have the same number of blocks. Additional experiments with other architectures are provided in Appendix. Datasets. We performed experiments on CIFAR-10 , LSUN-bedroom and Anime Faces datasets. For different datasets we use different numbers of discriminator steps per one generator step d steps and different numbers of blocks in a bucket n blocks. We summarize the main parameters used for three datasets in Table 1. In the last column we also report Coverage, which is the ratio of the latent space cardinality (which equals the number of buckets to the power n blocks) to the dataset size. Intuitively, large coverage guarantees that RPGAN has a sufficiently rich latent space of generator routes to capture the reference dataset. In the experiments below, we demonstrate that even moderate coverage is sufficient to generate high-fidelity images (see the LSUN-bedroom dataset with coverage ≈ 3.3). Training details. We use the Adam optimizer with learning rate equal to 0.25 × 10 −3, β 1, β 2 equal to 0.5, 0.999 and train the model for 45 × 10 4 generator steps for CIFAR-10 and 25 × 10 4 generator steps for Anime Faces and LSUN-bedroom datasets. During training we also learn the unique input vector Z. We observed that a learnable Z slightly improves the final generation quality and stabilizes the learning process. During the training step, we pass Z through N independent random paths. Formally, let {x 1, . . ., x N} be a batch of samples received from a bucket B k. To pass this batch through the bucket B k+1 we take random blocks B ki1,..., B ki N and form a new batch In all the experiments, we use the same training protocols for both the RPGAN and the standard GAN of the generator architecture. Note, that despite larger number of learnable parameters, RPGAN does not require more data or training time to achieve the same quality, compared to standard GANs. Figure 3: Top image: sample generated with a fixed sequence of blocks. Horizontal lines: samples generated by the same sequence of blocks in all buckets and one unfrozen bucket. In the selected bucket we choose ten arbitrary blocks to avoid excessively large figures. The samples produced in this way allow to interpret factors of variation, captured by different buckets. In the first series of experiments we investigate the " responsibility areas" of different generator layers. This can be performed with a technique schematically presented on Figure 2. The goal is to interpret the role of the third bucket B 3 in a five-bucket generator. For all other buckets B 1, B 2, B 4, B 5 we fix arbitrary blocks, shown in blue on Figure 2. Then we generate images corresponding to routes that contain all the fixed blocks, with the stochasticity coming only from varying blocks inside the target bucket B 3. By inspecting the distribution of the obtained images, we can understand what factors of variation are influenced by B 3. On Figure 3 we plot the samples generated with only one "unfrozen" bucket for one image from the CIFAR-10 dataset. Thus, each row shows how the original generated image could change if different blocks from the corresponding bucket are used. Several observations from Figure 3, Figure 9, Figure 12 and Figure 15 are listed below. The first bucket typically does not influence coloring and mostly affects small objects' deformations. The intermediate buckets have the largest influence on semantics. The last two buckets are mostly responsible for coloring and do not influence the content shape. In particular, on Figure 3 the fourth layer widely varies color, while the fifth acts as a general tone corrector. Note that these findings are consistent with the insights revealed by. To confirm the observations quantitatively, we perform the following experiment. We define a metric d img that evaluates the similarity between two generated images. Different metrics are able to capture different variations (e.g. in terms of semantic, color histogram, etc.) and we describe two particular choices of d img below. Then we choose a random route in the RPGAN generator and for each bucket 4, varying blocks in B l. In other words, rather then taking all possible pairs, we take four random samples from each line of the table in the Figure 3. Then we measure diversity w.r.t. d img captured by bucket B l as a ratio Intuitively, we compute the relative diversity with respect to the first layer, which typically captures the smallest amount of variations in our experiments. We then average these ratios over 100 independent evaluations. High values of averaged ratio D l→1,d img imply higher diversity of the bucket B l compared to the first bucket in terms of the metric d img. For d img we experimented with the following two metrics, capturing semantics and color differences correspondingly. Inspired by the well-known Fréchet Inception Score concept , we consider the Euclidean distance between the outputs of the last layer of the pretrained InceptionV3 network for semantic distance evaluation. Namely, we define d semantic (img 1, img 2) as Iv3(img 1) − Iv3(img 2) 2 were Iv3(img) is the InceptionV3 model activations for image img. To measure differences in color, we take the Hellinger distance between the color histograms of generated samples. Namely, for each color channel, we split the range [0, . . ., 255] into 25 equal segments and evaluate the discrete distribution defined by the frequencies the sample's pixel intensities appear in a given segment. Then the Hellinger distance between two quantified color distributions is defined as We compute this metric for each color channel independently. The average values of D l→1,d img with the standard deviations are shown on Figure 4. It demonstrates that the semantic diversity is the largest for the intermediate layers. On the contrary, the last buckets, which are closer to the output, do not influence semantics but have higher impact in terms of color. Note that the first layer always shows the smallest variability in terms of both semantics and colors. The last bucket seems to be responsible for color correction and color inversion and has a lower pallet variability impact. Note, that the plots from Figure 4 reaffirm the findings coming from the Figure 3. Note that the similar empirical also hold for other datasets (see figures 13, 16 in appendix). Overall, we summarize the main findings common for CIFAR-10, LSUN and Anime Faces datasets as: • The earlier layers have a smaller variability and seem to be responsible for the viewpoint and the position of the object on the image. • The semantic details of the image content are mostly determined by the intermediate layers. • The last layers typically affect only coloring scheme and do not affect content semantics or image geometry. Note, that these can differ for other datasets or other generator architectures. For instance, for the four-bucket generator and MNIST (Figure 6, left) or randomly colored MNIST (Figure 8, left) the semantics are mostly determined by the first two buckets. In this subsection, we argue that the interpretations of different layers, obtained with RPGAN, are also valid for the standard GAN generator of the same architecture. First, we demonstrate that both standard GAN and RPGAN trained under the same training protocol provide almost the same generation quality. As a standard evaluation measure, we use the Fréchet Inception Distance (FID) introduced in. We also compute precision-recall curve as defined in. For evaluation on CIFAR-10, we use 50000 generated samples and the whole train dataset. We also take ten independently trained generators and report minimal and average FID values. See Table 2 for FID comparison and Figure 11 in appendix for precision-recall. RPGAN and SN-ResNet perform with the same quality both in terms of FID and precision-recall curves. Table 2: FID values for CIFAR-10. To confirm that the layers of the standard GAN generator can be interpreted in the same way as the corresponding layers of its RPGAN counterpart, we perform the following experiment. We take a standard SN-ResNet GAN, consisting of five layers associated with the correspondent buckets in RPGAN, and train it on CIFAR-10. Then for each layer, we add normal noise to its weights. Intuitively, we expect that the noise injection in the particular layer would change generated samples in terms of characteristics, influenced by this layer. For instance, noise in the last two layers is expected to harm coloring scheme, while noise in the intermediate layers is expected to bring maximal semantic damage. Several samples, produced by perturbed layers, are presented on Figure 5. The images support the intuition described above and confirm that RPGAN may serve as an analysis tool for the corresponding generator model. Note, however, that injecting noise per se is not sufficient for interpretability. The perturbed generators produce poor images, which are difficult to analyze. Meanwhile, RPGAN always generates good-looking images, which allows to identify the factors of variation, corresponding to the particular layer. For instance, see Figure 8 for the colored MNIST dataset. Figure 8 (left) shows plausible images, generated by varying RPGAN blocks. In contrast, Figure 8 (right) demonstrates images from generators perturbed with small and large noise. For both noise magnitudes, these images are difficult to interpret. Of course, given the interpretations obtained via RPGAN, one can perceive similar patterns in the noisy generations, but noise injection alone is not sufficient for interpretability. In the next experiment, we demonstrate that the RPGAN model is also a natural fit for the generative incremental learning task (see e.g., are common for D 1 and D 2 . To illustrate this scenario, we take a partition of the MNIST handwritten digits dataset into two subsets MNIST 0−6 and MNIST 7−9 of digits from 0 to 6 and from 7 to 9 correspondingly. As for generator for MNIST 0−6 we take a 4-bucket RPGAN model with a number of blocks equal to 20, 20, 20, 8. Note that the last bucket is much thinner than the others, as it turns out to be responsible for variations in writing style, which does not change much across the dataset. Then we train the generator on the subset MNIST 0−6 of first 7 digits (see Figure 6, left and center). After that we add five additional blocks to each of the first two layers, obtaining a generator with a number of blocks equal to 25, 25, 20, 8 and pretrained weights in all blocks but five in the first and in the second buckets. Then we train the extended model to fit the whole MNIST by optimizing only the ten new blocks (see Figure 6, right). As a surprising side effect of our model, we discovered that decent generation quality can be achieved by the RPGAN generator with no nonlinearities, i.e., one can train the RPGAN generator with all blocks consisting of linear transformations only. To demonstrate that, we take an RPGAN with the same ResNet-like generator architecture as in the experiments above. Then we replace all nonlinearities in the generator model by identity operations and train it on the CIFAR-10 dataset. The model demonstrates FID equal to 22.79 that is competitive to the state-of-the-art generative models of comparable sizes (see Figure 23 for generated images examples). Note that this approach fails for a standard GAN generator that maps a Gaussian distribution to an images distribution. Indeed, that generator would be a linear operator from a latent space to the images space with a Gaussian distribution in the images domain. This purely linear generator architecture allows us to significantly speed up the image generation process for fully-connected layers. We group consequent buckets of fully-connected layers to form a new bucket. Blocks in the new bucket are linear transformations that are products of the blocks from the original buckets. To demonstrate this, we train a fully-connected generator network on the MNIST dataset (see Table 3 in appendix, left column). Then we join the last three buckets into a single one. We form a new bucket by blocks defined as the linear operators B 5k • B 4j • B 3i where i, j, k are random indices of blocks from the buckets B 3, B 4, B 5 of the original generator. Thus instead of performing three multiplications of features vector from the second layer by matrices of the shapes 256 × 512, 512 × 1024, 1024 × 784 we perform a single multiplication by a 256 × 784 matrix. In our experiments, we achieved ×2.2 speed up. Note, however, that after the compression, the latent space cardinality can decrease if a small subset of tuples (i, j, k) is used to populate the new bucket. Nevertheless, as random products of joining buckets are used, we expect that the generated images would be uniformly distributed in the space of images, produced by the uncompressed generator (see Figure 24 for samples comparison). In this paper, we address the interpretability of generative models. In particular, we have introduced RPGAN, an alternative design of generative adversarial networks, which allows natural interpretation of different generator layers via using random routing as a source of stochasticity. With experiments on several datasets, we provide evidence that different layers are responsible for the different factors of variation in generated images, which is consistent with findings from previous work. As a possible direction of future research, one can use the RPGAN analysis to construct efficient models, e.g., via identification of redundant parts of the generator for pruning or inference speedup. If the number of blocks is too low, the ing latent space appears to have insufficient cardinality to cover the dataset. On the other hand, a too high number of blocks in a difficult training procedure and also fails. To generate LSUN-bedroom-like images, we use the 7-bucket RPGAN with ResNet-like generator with five residual blocks, the first fully-connected layer, and the last convolutional layer. Similarly to CIFAR-10 experiments, during the generation, we freeze a random path and vary blocks in a single bucket to investigate its responsibility. See Figure 12 for blocks variations and Figure 13 for buckets responsibility analysis. Note that similarly to CIFAR-10, the central buckets have a maximal semantic impact. Last two buckets are mostly responsible for coloring. The first two buckets are responsible for local geometrical features. Note that here we face mode collapse for the third bucket: mainly, it affects only tiny local features. See Figure 14 for samples generated by the model. l o w n o i s e v a r i a n c e l a r g e n o i s e v a r i a n c e Figure 8: Left: images produced by varying blocks in a particular bucket of RPGAN. Right: images produced by the standard GAN after parameters perturbation in a particular generator layer, with low and high normal noise variance. A.3 ANIME FACES DATASET Though this dataset is not standard for GANs, we use it in the experiments as it nicely reveals the RPGAN analysis tools. Here we use the 6-bucket RPGAN with ResNet-like generator with four residual blocks, the first fully-connected layer, and the last convolutional layer. See Figure 15 for block variations and Figure 16 for bucket responsibility analysis. Again, the content semantics is mostly defined by the intermediate buckets. The last two buckets are mostly responsible for coloring: the fifth bucket has the maximal impact on coloring, and the last bucket varies tones. The first buckets are responsible for small details (one can note the hair on the character's forehead). See Figure 17 for samples generated by the model. Here we show that the concept of RPGAN works well with different generator architectures and learning strategies. Here we present plots for DCGAN-like generators consisted of consequent convolutional layers without skip connections. All the models were trained with the same parameters as described in Section 4. Despite of Spectral Normalization, we train these models as WGANs with weight penalty . On the Figure 19 we show plots for a four-bucket generator trained on CIFAR-10. We also train four-bucket generator on colored MNIST, see Figure 8, left. Finaly, we show plots for the five-bucket generator and CelebA-64x64 dataset on Figure 21. See Figure 18, Figure 21, Figure 22 for buckets analysis. In this section we show that injecting noise in the generator weights cannot be used as a stand-alone interpretability method. Namely, we compare images produces by RPGAN and noise injection for models trained on randomly colored MNIST samples. We train both RPGAN and standard generators as a Wasserstein GAN with weights penalty. Tanh, reshape to 28 × 28

Represent the short summary for finding the corresponding scientific article (long)

We introduce an alternative GAN design based on random routes in generator, which can serve as a tool for generative models interpretability.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.