Question: Represent the article for finding a single-sentence TLDR

Question: Off-policy reinforcement learning algorithms promise to be applicable in settings where only a fixed data-set (batch) of environment interactions is available and no new experience can be acquired. This property makes these algorithms appealing for real world problems such as robot control. In practice, however, standard off-policy algorithms fail in the batch setting for continuous control. In this paper, we propose a simple solution to this problem. It admits the use of data generated by arbitrary behavior policies and uses a learned prior -- the advantage-weighted behavior model (ABM) -- to bias the RL policy towards actions that have previously been executed and are likely to be successful on the new task. Our method can be seen as an extension of recent work on batch-RL that enables stable learning from conflicting data-sources. We find improvements on competitive baselines in a variety of RL tasks -- including standard continuous control benchmarks and multi-task learning for simulated and real-world robots. Batch reinforcement learning (RL) is the problem of learning a policy from a fixed, previously recorded, dataset without the opportunity to collect new data through interaction with the environment. This is in contrast to the typical RL setting which alternates between policy improvement and environment interaction (to acquire data for policy evaluation). In many real world domains collecting new data is laborious and costly, both in terms of experimentation time and hardware availability but also in terms of the human labour involved in supervising experiments. This is especially evident in robotics applications (see e.g. Haarnoja et al. 2018b; Kalashnikov et al. 2018 for recent examples learning on robots). In these settings where gathering new data is expensive compared to the cost of learning, batch RL promises to be a powerful solution. There exist a wide class of off-policy algorithms for reinforcement learning designed to handle data generated by a behavior policy µ which might differ from π, the policy that we are interested in learning (see e.g. for an introduction). One might thus expect solving batch RL to be a straightforward application of these algorithms. Surprisingly, for batch RL in continuous control domains, however, found that policies obtained via the naïve application of off-policy methods perform dramatically worse than the policy that was used to generate the data. This highlights the key challenge in batch RL: we need to exhaustively exploit the information that is in the data but avoid drawing for which there is no evidence (i.e. we need to avoid over-valuing state-action sequences not present in the training data). As we will show in this paper, the problems with existing methods in the batch learning setting are further exacerbated when the provided data contains behavioral trajectories from different policies µ 1,..., µ N which solve different tasks, or the same task in different ways (and thus potentially execute conflicting actions) that are not necessarily aligned with the target task that π should accomplish. We empirically show that previously suggested adaptations for off-policy learning can be led astray by behavioral patterns in the data that are consistent (i.e. policies that try to accomplish a different task or a subset of the goals for the target task) but not relevant for the task at hand. This situation is more damaging than learning from noisy or random data where the behavior policy is sub-optimal but is not predictable, i.e. the randomness is not a correlated signal that will be picked up by the learning algorithm. We propose to solve this problem by restricting our solutions to'stay close to the relevant data'. This is done by: 1) learning a prior that gives information about which candidate policies are potentially supported by the data (while ensuring that the prior focuses on relevant trajectories), 2) enforcing the policy improvement step to stay close to the learned prior policy. We propose a policy iteration algorithm in which the prior is learned to form an advantage-weighted model of the behavior data. This prior biases the RL policy towards previously experienced actions that also have a high chance of being successful in the current task. Our method enables stable learning from conflicting data sources and we show improvements on competitive baselines in a variety of RL tasks -including standard continuous control benchmarks and multi-task learning for simulated and real-world robots. We also find that utilizing an appropriate prior is sufficient to stabilize learning; demonstrating that the policy evaluation step is implicitly stabilized when a policy iteration algorithm is used -as long as care is taken to faithfully evaluate the value function within temporal difference calculations. This in a simpler algorithm than in previous work . In the following we consider the problem of reinforcement learning, modeling the environment as a markov decision process (MDP) consisting of the continuous states s ∈ S, actions a ∈ A, and transition probability distribution p(s t+1 |s t, a t) -describing the evolution of the system dynamics over time (e.g. probability of reaching s t+1 from state s t when executing action a t) -together with the state-visitation distribution p(s). The goal of reinforcement learning is to find a policy π(a|s) that maximizes the cumulative discounted return, for the reward function r(s, a) ∈ R. We also define the state-action value function for taking action a t in state s t, and thereafter following π:, which we can relate to the objective J via J(π) =, where π * is the optimal policy. We parameterize the policy π θ (a|s) by θ but we will omit this dependency where unambiguous. In some of the experiments we will also consider a setting where we learn about multiple tasks k ∈ {1, . . . K}, each with their own reward function r k (s, a). We condition the policy and Q-function on the task index k (i.e. Q π k and π(a|s, k)), changing the objective to maximize the sum of returns across all tasks. For the batch RL setting we assume that we are given a dataset D µ containing trajectory snippets (i.e. sub-trajectories of length N) τ = {(s 0, a 0), · · ·, (s T, a T)}, with τ ∈ D µ. We assume access to the reward function r for the task of interest and can evaluate it for all transitions in D µ (for example, r may be some function of s). We further assume D µ was filled, prior to training, by following a set of arbitrary N behavior policies µ 1, · · ·, µ N. Note that these behavior policies may try to accomplish the task we are interested in; or might indeed generate trajectories unrelated to the task at hand. To stabilize off-policy RL from batch data, we want to restrict the learned policy to those parts of the state-action space supported by the batch. In practice this means that we need to approximately restrict the policy to the support of the empirical state-conditional action distribution. This prevents the policy from taking actions for which the Q-function cannot be trained and for which it might thus give erroneous, overly optimistic values . In this paper we achieve this by adopting a policy iteration procedure -in which the policy is constrained in the improvement step. As in standard policy iteration , the procedure consists of two alternating steps. First, starting with a given policy π i = π θi in iteration i (with π θ0 corresponding to a randomly initialized policy distribution), we find an approximate action-value function Q πi (s, a) ≈Q(s, a; φ i), with parameters φ (Section 3.1) (as with the policy we will drop the dependence on φ i and writeQ πi (s, a) where unambiguous). Second, we optimize for π i+1 with respect toQ πi subject to a constraint that ensures closeness to the empirical state-conditional action distribution of the batch (Section 3.2). Iterating these steps, overall, optimizes J(π). We realize both policy evaluation and improvement via a fixed number of gradient descent steps -holding π andQ πi fixed via the use of target networks . We refer to Algorithm 1 for details. To learn the task action-value function in each iteration we minimize the squared temporal difference error for a given reward -note that when performing offline RL from a batch of data the reward r might be computed post-hoc and does not necessarily correspond to the reward optimized by the behavior policies µ 1, · · ·, µ N. The after iteration i is given as We approximate the expectation required to calculateV πi (s) with M samples from As further discussed in the related work Section 4, the use of policy evaluation is different from the Q-learning approach pursued in; , which requires a maximum over actions and may be more susceptible to overestimation of Q-values. We find that when enough samples are taken (we use M = 20) and the policy is appropriately regularized (see Section 3.2) learning is stable without additional modifications. In the policy improvement step we solve the following constrained optimization problem where D µ is the behavior data, π the policy being learned, and π prior is the prior policy. This is similar to the policy improvement step in but instead of enforcing closeness to the previous policy here the constraint is with respect to a separately learned "prior" policy, the behavior model. The role of π prior in Equation 2 is to keep the policy close to the regime of the actions found in D µ. We consider two different ways to express this idea by learning a prior alongside the policy optimization. It should be noted that, if the prior itself is already good enough to solve the task we can set π i+1 = π prior (corresponding to = 0) -i.e. if the data stems from an expert and the prior takes the learned Q-values into account -skipping the policy improvement step and further simplifying the algorithm. For learning the prior, we first consider simply modeling the raw behavior data. This is similar to the approach of BCQ and BEAR-QL , but we use a parametric behavior model and measure distance by KL; we refer to the related work for a discussion. The behavior model can be learned by maximizing the log likelihood of the observed data where θ bm are the parameters of the behavior model prior. Regularizing towards the behavior model can help to prevent the use of unobserved actions, but it may also prevent the policy from improving over the behavior in D µ. In effect, the simple behavior prior in Equation 3 regularizes the new policy towards the empirical state-conditional action distribution in D µ. This may be acceptable for datasets dominated by successful trajectories for the task of interest or when the unsuccessful trajectories are not predictable (i.e. they correspond to random behaviour). However, we here are interested in the case where D µ is collected from imperfect data and from multiple tasks. In this case, D µ will contain a diverse set of trajectories -both (partially) successful and actively harmful for the target task. With this in mind, we consider a second learned prior, the advantage-weighted behavior model, π abm, with which we can bias the RL policy to choose actions that are both supported by D µ and also good for the current task (i.e. keep doing actions that work). We can formulate this as maximizing the following objective: where f is an increasing, non-negative function, and the difference R (τ t:N) −V πi is akin to an n-step advantage function, but here calculated off-policy representing the "advantage" of the behavior snippet over the policy π i. This objective still tries to maximize the log likelihood of observed actions, and avoids taking actions not supported by data. However, by "advantage weighting" we focus the model on "good" actions while ignoring poor actions. We let f = 1 + (the unit step function with f (x) = 1 for x ≥ 0 and 0 otherwise) both for simplicty -to keep the number of hyperparameters to a minimum while keeping the prior broad -and because it has an intuitive interpretation: such a prior will start by covering the full data and, over time, filter out trajectories that would lead to worse performance than the current policy, until it eventually converges to the best trajectory snippets contained in the data. We note that Equation 4 is similar to a policy gradient, though samples here stem from the buffer and it will thus not necessarily converge to the optimal policy in itself; π θabm will instead only cover the best trajectories in the data due to no importance weighting being performed for off-policy data. This bias is in fact desirable in a batch-RL setting; we want a broad prior that only considers actions present in the data. We also note that we tried several different functions for f including exponentiation, e.g. f (x) = exp(x), but found that choice of function did not make a significant difference in our experiments. Using either π θbm or π θabm as π prior, Equation 2 can be solved with a variety of optimization schemes. We experimented with an EM-style optimization following the derivations for the MPO algorithm, as well as directly using the stochastic value gradient ofQ πi wrt. policy parameters . We can optimize the objective from Equation 2 using a two-step procedure. Following, we first notice that the optimal π for Equation 2 can be expressed asπ(a|s) ∝ π prior (a|s) exp(Q π i (s,a) /η), where η is a temperature that depends on the used for the KL constraint and can be found automatically by a convex optimization (Appendix B.1). Conveniently, we can sample from this distribution by queryingQ πi using samples from π prior. These samples can then be used to learn the parametric policy by minimizing the divergence KL(π π θi+1), which is equivalent to maximizing the weighted log likelihood which we optimize via gradient descent subject to an additional trust-region constraint on π θ given as KL(π θi π θ) < trust to ensure conservative updates (Appendix B.1). Stochastic value gradient optimization Alternatively, we can use Langrangian relaxation to turn Equation 2 into an objective amenable to gradient descent. Inserting π θ for π and relaxing in for η > 0 and which we can optimize by alternating gradient descent steps on θ and η respectively, taking the stochastic gradient of the Q-value through the sampling of a ∼ π θ (·, s) via re-parameterization. See Appendix B.2 for a derivation of this gradient. Full pseudocode for our approach can be found in Appendix A, Algorithm 1. There exist a number of off-policy RL algorithms that have been developed since the inception of the RL paradigm (see e.g. for an overview). Most relevant for our work, some of these have been studied in combination with function approximators (for estimating value functions and policies) with an eye on convergence properties in the batch RL setting. In particular, several papers have theoretically analyzed the accumulation of bootstrapping errors in approximate dynamic programming and approximate policy iteration ; for the latter of which there exist well known algorithms that are stable at least with linear function approximation (see e.g.). Work on RL with non-linear function approximators has mainly considered the "online" or "growing batch" settings, where additional exploration data is collected (; ;); though some success for batch RL in discrete domains has been reported . For continuous action domains, however, off-policy algorithms that are commonly used with powerful function approximators fail in the fixed batch setting. Prior work has identified the cause of these failures as extrapolation or bootstrapping errors which occur due to a failure to accurately estimate Q-values, especially for state-action pairs not present in the fixed data set. Greedy exploitation of such misleading Q-values (e.g. due to a max operation) can then cause further propagation of such errors in the Bellman backup, and to inappropriate action choices during policy execution (leading to suboptimal behavior). In non-batch settings, new data gathered during exploration allows for the Q-function to be corrected. In the batch setting, however, this feedback loop is broken, and correction never occurs. To mitigate these problems, previous algorithms based on Q-learning identified two potential solutions: 1) correcting for overly optimistic Q-values in the Bellman update, and 2) restricting the policy from taking actions unlikely to occur in the data. To address 1) prior work uses a Bellman backup operator in which the max operation is replaced by a generative model of actions which a learned policy is only allowed to minimally perturb; or via a maximum over actions sampled from a policy which is constrained to stay close to the data (implemented through a constraint on the distance to a model of the empirical data, measured either in terms of maximum mean discrepancy or relative entropy). To further penalize uncertainty in the Q-values this can be combined with Clipped Double-Q learning or an ensemble of Q-networks . To address 2) prior work uses a similarly constrained max also during execution, by considering only actions sampled from the perturbed generative model or the constrained policy , and choosing the best among them. Our work is based on a policy iteration scheme instead of Q-learning -exchanging the max for an expectation. Thus we directly learn a parametric policy that we also use for execution. We estimate the Q-function as part of the policy evaluation step with standard TD-0 backups. We find that for an appropriately constrained policy no special treatment of the backup operator is necessary, and that it is sufficient to simply use an adequate number of samples to approximate the expectation when estimating V (see Equation 1). The only modification required is in the policy improvement step where we constrain the policy to remain close to the adaptive prior in Equation 2. As we demonstrate in the empirical evaluation it is the particular nature of the adaptive prior -which can adapt to the task at hand (see Equation 4) -that makes this constraint work well. Additional measures to account for uncertainty in the Q values could also be integrated into our policy evaluation step but we did not find it to be necessary for this work; we thus forego this in favor of our simpler procedure. Our policy iteration scheme also bears similarity to previous works that use (relative) entropy regularized policy updates which implement constraints with respect to either a fixed (e.g. uniform) policy (e.g. a) or, in a trust-region like scheme, to the previous policy (e.g. . Other work has also focused on policy priors that are optimized to be different from the actual policy but so far mainly in the multi-task or transfer-learning setup, i.e. to share knowledge across or to transfer knowledge to new tasks . The constrained updates are also related to trustregion optimization in action space, e.g. in TRPO / PPO (; and MPO, which ensures stable learning in the standard RL setting by enforcing conservative updates. The idea of conservative policy optimization can be traced back to . Here we take a slightly different perspective: we enforce a trust region constraint not on the last policy in the policy optimization loop (conservative updates) but wrt. the advantage weighted behavior distribution. Figure 2: Control suite when using only the first 2k episodes of low-quality data from each run; return for best behavior episode in the data was 718/159/435/728 (left to right). While plain RL is able to learn on walker, learned priors improve performance and stabilize learning for cheetah and quadruped (with overall lower performance than when learning from good data). We experiment with continuous control tasks in two different settings. In a first set of experiments we compare our algorithm to strong off-policy baselines on tasks from the DeepMind control suite -to give a reference point as to how our algorithm performs on common benchmarks. We then turn to the more challenging setting of learning multiple tasks involving manipulation of blocks using a robot arm in simulation. These span tasks from reaching toward a block to stacking one block on top of another. Finally, we experiment with analogous tasks on a real robot. We use the same networks for all algorithms that we compare, optimize parameters using Adam , and utilize proprioceptive features (e.g. joint positions / velocities) together with task relevant information (mujoco state for the control suite, and position/velocity estimates of the blocks for the manipulation tasks). All algorithms were implemented in the same framework, including our reproduction of BCQ and BEAR, and differ only in their update rules. Note that for BEAR we use a KL instead of the MMD as we found this to work well, see appendix. In the multi-task setting (Section 5.1) we learn a task conditional policy π θ (a|s, k) and Q-functionQ π φ (s, a, k) where k is a one-hot encoding of the task identifier, that is provided as an additional network input. We refer to the appendix for additional details. We start by performing experiments on four tasks from the DeepMind control suite: Cheetah, Hopper, Quadruped. To obtain data for the offline learning experiments we first generate a fixed dataset via a standard learning run using MPO, storing all transitions generated; we repeat this with 5 seeds for each environment. We then separate this collected data into two sets: for experiments in the high data regime, we use the first 10,000 episodes generated from each seed. For experiments with low-quality data we use the first 2,000 episodes from each seed. The high data regime therefore has both more data and data from policies which are of higher quality on average. A plot showing the performance of the initial training seeds over episodes is given in the appendix, Figure 6. Bring to corner ABM+MPO BM+MPO MPO Figure 4: (Left) Learning curves for the tasks "bring to corner" and "bring to center". These tasks were learned using only data from the seven intial stacking tasks. The stacking dataset was rich enough to learn these new tasks fully offline with ABM. (Right) Simulated Sawyer environment. For our offline learning experiments, we reload this data into a replay buffer. 1 The dataset is then fixed and no new transitions are added; the offline learner never receives any data that any of its current or previous policies have generated. We evaluate performance by concurrently testing the policy in the environment. The of this evaluation are shown in Figure 1. As can be observed, standard off-policy RL algorithms (MPO / SVG) can learn some tasks offline with enough data, but learning is unstable even on these relatively simple control suite tasks -confirming previous findings from . In contrast the other methods learn stably in the high-data regime, with BCQ lagging behind in Hopper and Quadruped (sticking too close to the VAE prior actions, an effect already observed in ). Remarkably our simple method of combining a policy iteration loop with a behavior model prior (BM+MPO in the plot) performs as well or better than the more complex baselines (BEAR and BCQ from the literature). Further improvement can be obtained using our advantage weighted behavior model even in some of these simple domains (ABM+SVG and ABM+MPO). Comparing the performance of the priors (BM [prior] vs ABM [prior], dotted lines) on Hopper we can understand the advantage that ABM has over BM: the BM prior performs well on simple tasks, but struggles when the data contains conflicting trajectories (as in Hopper, where some of the seeds learn sub-optimal jumping) leading to too hard constraints on the RL policy. Interestingly, the ABM prior itself performs as well or better than the baseline methods for the control-suite domains. Furthermore, in additional experiments presented in the appendix we find that competitive performance can be achieved, in simple domains, when training only an ABM prior (effectively setting = 0); providing an even simpler method when one does not care about squeezing out every last bit of performance. A test on lower quality data, Figure 2, shows similar trends, with our method learning to perform slightly better than the best trajectories in the data. We experiment with a Sawyer robot arm simulated in Mujoco in a multi-task setting -as described above. The seven tasks are to manipulate blocks that are placed in the workspace of the robot. They include: reaching for the green block (Reach), grasping any block (Grasp), lifting the green block (Lift), hovering the green block over the yellow block (Place Wide), hovering the green block over the center of the yellow block (Place Narrow), stacking the green block on top of yellow (Stack and Stack/Leave i.e. without gripper contact). To generate the data for this experiment we again run MPO -here simultaneously learning all task-conditional policies for the full seven tasks. Data-was collected by randomly switching tasks after each episode (of 200 control steps) with random resets of the robot position every 20 episodes. As before, data from all executed tasks is collected in one big data-set annotating each trajectory snippet with all rewards (i.e. this is similar to the SAC-R setting from). During offline learning we then compare the performance of MPO and RL with a behavior modelling prior (BM+MPO and ABM+MPO). As shown in Figure 3, behavioral modelling priors improve performance across all tasks over standard MPO -which struggles in these more challenging tasks. This is likely due to the sequential nature of the tasks: later tasks implicitly include earlier tasks but only a smaller fraction of trajectories achieve success on stack and leave (and the actions needed for stack conflict, e.g., with lifting), this causes the BM prior to be overly broad (see plots in appendix). The ABM+MPO, on the other hand, achieves high performance across all tasks. Interestingly, even with ABM in place, the RL policy learned using this prior still outperforms the prior, demonstrating that RL is still useful in this setting. As an additional experiment we test whether we can learn new tasks entirely from previously recorded data. Since our rewards are specified as functions of observations, we compute rewards for two new tasks (bringing the green block to the center and bringing it to the corner) for the entire datasetwe then test the ing policy in the simulator. As depicted in Figure 4, this is successful with ABM+MPO, demonstrating that we can learn tasks which were not originally executed in the dataset (as long as trajectory snippets that lead to successful task execution are contained in the data). Finally, to validate that our approach is a feasible solution to performing fast learning for real-robot experiments, we perform an experiment using a real Sawyer arm and the same set of seven tasks (implemented on the real robot) from Section 5.2. As before, data from all executed tasks is collected in one big data-set annotating each trajectory snippet with all rewards. The full buffer after about two weeks of real robot training is used as the data for offline learning; which we here only performed with ABM+MPO due to the costly evaluation. The goal is to re-learn all seven original tasks. Figure 5 shows the of this experiment -we ran an evaluation script on the robot, continuously testing the offline learned policy, and stopped when there was no improvement in average reward (as measured over a window of 50 episodes). As can be seen, ABM with MPO as the optimizer manages to reliably re-learn all seven tasks purely from the logged data in less than 12 hours. All tasks can jointly be learned with only small differences in convergence time -while during the initial training run the harder tasks, of course, took the most time to learn. This suggests that gathering large data-sets of experience from previous robot learning experiments, and then quickly extracting the skills of interest, might be a viable strategy for making progress in robotics. In this work, we considered the problem of stable learning from logged experience with off-policy RL algorithms. Our approach consists of using a learned prior that models the behavior distribution contained in the data (the advantage weighted behavior model) towards which the policy of an RL algorithm is regularized. This allows us to avoid drawing for which there is no evidence in the data. Our approach is robust to large amounts of sub-optimal data, and compares favourably to strong baselines on standard continuous control benchmarks. We further demonstrate that our approach can work in challenging robot manipulation domains -learning some tasks without ever seeing a single trajectory for them. A ALGORITHM A full algorithm listing for our procedure is given in Algorithm 1. Input: N steps number of learning steps, N TU steps between target update, M number of action samples, KL regularization parameter, initial parameters for θ, η, α and φ initialize N = 0, θ = θ, φ = φ while i ≤ N steps do sample a batch B of trajectories τ from replay buffer D µ sample M actions from policies to estimate expectations below // compute gradient for prior model For BM prior: δ θprior ← ∇ θprior τ ∈B (st,at)∈τ log π θbm (a t |s t) For ABM prior: δ θprior ← ∇ θabm τ ∈B (st,at)∈τ 1 + (R(τ 1:|τ |) −V φ (s t)) log π θabm (a t |s t) // compute gradients for Q, π and η δ φ ← ∇ φ τ ∈B i∼I (st,at)∈τ r(s t, a t) + γV φ (s t) −Q φ (s t, a t) 2 For MPO: We here give additional details on the implementation of the policy improvement step in our algorithm. Depending on the policy optimizer used (MPO or SVG) different update rules are used to maximize the objective given in Equation 4. This is also outlined in Algorithm 1. We here describe the general form for both algorithms, using π prior to represent the prior which then will be instantiated as either the ABM or BM prior. For the EM-style optimization based on MPO we first notice that the optimal non-parametric policy that respects the KL constraint wrt. π prior iŝ π(a|s) = π prior (a|s) exp(Q π i (s,aj) /η − log Z), with Z = π prior (a|s) exp(Q π i (s,a) /η)da and where η is a temperature that depends on the desired constraint. In practice we estimate Z based on the M samples {a 1, . . ., a M} ∼ π prior (a|s) that we draw for each state to perform the optimization of π θ. That is we set Z ≈ 1 /M M j=1 exp(Q π i (s,aj) /η). Using these samples we can then optimize for η in a way analogous to what is described in. Specifically, we find that the objective for finding η is which can approximate based on a batch B of trajectories sampled from D µ (sampling M actions from π prior for each state therein) and corresponding action samples where we used the samples which can readily be differentiated wrt. η. We then use Adam (with standard settings and learning rate 2e − 4) to take a gradient step in direction of ∇ η g(η) (we want to maximize g(η)) for each batch. We start our optimization with η = 3 to ensure stable optimization (i.e. to avoid large changes in the policy parameters in the beginning of optimization). Further, after each gradient step, we project η to the positive numbers i.e. we set η = max(η, 0.001), as η is required to be positive. We find that this procedure is capable of fulfilling the desired KL constraints well. The same batch, and action samples, are then also used to take an optimization step for the policy parameters θ. In particular we find the parametric policy by minimizing the divergence KL(π π θi+1), which is equivalent to maximizing the weighted log likelihood of sampled actions (Equation 5). We only take M = 20 samples here, which is a relatively crude representation of the behavior model at state s; therefore, to prevent the policy from converging too quickly it can be useful to employ an additional trust region constraint in this step that ensures slow convergence. We do this by adjusting the maximum likelihood objective Equation 5 to contain an additional KL regularization towards the previous policy, yielding the following optimization problem using samples {a 1, . . ., a M} ∼ π prior (a|s): for α > 0, which is a Langrangian relaxation to the maximum likelihood problem under the additional constraint that KL(π θi π θ) < trust, where α is the Langrange multiplier. This objective can be differentiated wrt. both θ and α and we simply take alternating gradient descent steps (one per batch for both θ and α) using Adam (starting with a random θ 0 and α = 1) and projecting α back to the positive regime if it becomes negative; i.e. we set α = max(α, 0.001). To optimize the policy parameters θ via the stochastic value gradient (under a KL constraints) we can directly calculate the derivative of the Langrangian relaxation from Equation 6. In particular, again assuming that we have sampled a batch B, the gradient wrt. θ can be obtained via the reparameterization trick . For this we first require that a sample from our policy π θ (a|s) can be obtained via a deterministic function applied to a standard noise source. We first specify the policy class used in the paper to be that of Gaussian policies, parameterized as π θ (a|s) = N (a|µ θ (s), Iσ 2 θ (s)), where N (a|µ, Σ) denotes the pdf of a standard Normal distribution and where we assume µ θ is directly given as one output of the network whereas we parameterize the diagonal standard deviation as σ θ (s) = sof tplus(h θ (s)) with h θ (s) being output by the neural network. We can then obtain samples via the deterministic transformation f (s, ξ; θ) = µ θ (s) + σ θ (s)ξ, where ξ ∼ N (0, I) (I being the identity matrix). Using this definition we can obtain the following expression for the value gradient: where we use the Gaussian samples {ξ 1, . . ., ξ M} ∼ N (0, I). The gradient for the Langrangian multiplier is given as where we dropped terms independent of η in the second line. Following δ θ to maximize the objective, and conversely moving in the opposite direction of δ η to minimize the objective wrt. η, can then be performed by taking alternating gradient steps. We perform one step per batch for both η and θ via Adam; starting from an initially random θ and η = 1. As in the MPO procedure we ensure that η is positive by projecting it to the positive regime after each gradient step. Hyperparameters for our MPO, SVG, and BCQ single-task experiments are shown in tables 1, 2, and 3, respectively. For multitask experiments, we modify the parameters shown in 5. To provide strong off-policy learning baselines we re-implemented BCQ and BEAR in the same framework that we used to implement our own algorithm. As mentioned in the main paper we used the same network architecture for ll algorithms. Algorithm specific hyperparameters where tuned via a coarse grid search on the control suite tasks; while following the advice from the original papers on good parameter ranges. To avoid bias in our comparisons we did not utilize ensembles of Q-functions for any of the methods (e.g. we removed them from BEAR), we note that ensembling did not seem to have a major impact on performance (see appendix in ). Parameters for all methods where optimized with Adam. Furthermore, to apply BEAR and BCQ in the multi-task setting we employed the same conditioning of the policy and Q-function on a one-hot task vector (which is used to select among multiple network "heads", yielding per task parameters, see description below). For BCQ we used a range of [0.25, 0.25] for the perturbative actions generated by the DDPG trained network, and chose a latent dimensionality of 64 for the VAE, see Table 3 for the full hyperparameters. For BEAR we used a KL constraint rather than the maximum mean discrepancy. This ensures comparability with our method and we did not see any issues with instability when using a KL. To ensure good satisfaction of constraints we used the exact same optimization for the Langragian multiplier required in BEAR that was also used for our method -see description of SVG above. The hyperparameters for the BEAR training run on the control suite are given in The task setup for both the simulated and real robot experiments is described in the following. A detailed description of the robot setup will be given in an accompanying paper. We nonetheless give a description here for completeness. We make no claim to have contributed these tasks specifically for this paper and merely use them as an evaluation test-bed. As the robot we utilize a Sawyer robotic arm mounted on a table and equipped with a Robotiq 2F-85 parallel gripper. A basket is positioned in front of the robot which contains three cubes (the proportions of cubes and basket sizes are consistent between simulation and reality). Three cameras on the basket track the cube using augmented reality tags. To model the tasks as an MDP we provide both proprioceptive information from the robot sensors (joint positions, velocities and torques) and the tracked cube position, velocity (both in 3 dimensions) and orientation to the policy and Q-function. Replay buffer size 2 × 10 6 Batch size 512 Table 5: Network parameters for multitask experiments. "->" indicates the network branching into separate "heads" for each task, which are selected based on the one-hot task vector. This architecture is similar to the design presented in. Note that transitions are duplicated in replay for each task (so that transitions are sampled independently for each task). Where d(a, b) denotes the euclidean distance between a and b. We also define two tolerance functions with outputs scaled between 0 and 1, i.e, D ADDITIONAL EXPERIMENTAL We present additional plots that show some aspects of the developed algorithm in more detail. We provide expanded plots for MPO on the control suite in Figure 7. These show in detail that the learned advantage weighted behavior model (ABM, in red in the left column) is far superior to the standard behavior model prior, leading to less constrained RL policies. Figure 8 shows full for the simulated robot stacking task, including all 7 intentions as well as performance of the prior policies themselves during learning. The task set is structured: earlier tasks like reaching and lifting are necessary to perform most other tasks, so the simple behavioral model performs well on these. For the more difficult stacking tasks, however, the presence of conflicting data means the simple behavioral model doesn't achieve high reward, though it still significantly improves performance of the regularized policy. Table 6 shows final performance for all methods on control suite tasks. Table 7 shows final performance on simulated robotics tasks to make comparison between algorithms easier. The episode returns are averaged over the final 10% of episodes. ABM provides a performance boost over BM Table 7: Returns on each simulated robotics task -for comparison between algorithms. alone, particularly for difficult tasks such as block stacking and quadruped. The RL policy further improves performance on the difficult tasks. As noted in the main paper, one additional option, to further simplify the algorithm is to omit the policy improvement step, setting π i+1 = π prior (i.e. considering the case where = 0) and, conversely learning the Q-values of the prior. In additional experiments we have found that this procedure roughly recovers the performance of the ABM prior when trained together with MPO (ABM+MPO); i.e. this is an option for a simpler algorithm to implement at the cost of some performance loss (especially on the most complicated domains). We included this setting as ABM (= 0, |τ | = 2) in the table -noting that in this case it is vital to choose short trajectory snippets in order to allow π prior to pick the best action in each state.

Represent the short summary for finding the corresponding scientific article (long):

We develop a method for stable offline reinforcement learning from logged data. The key is to regularize the RL policy towards a learned "advantage weighted" model of the data.

We augment the Q-value estimates with a count-based bonus that ensures optimism during action selection and bootstrapping, even if the Q-value estimates are pessimistic.


Question: One of the main challenges in applying graph convolutional neural networks on gene-interaction data is the lack of understanding of the vector space to which they belong and also the inherent difficulties involved in representing those interactions on a significantly lower dimension, viz Euclidean spaces. The challenge becomes more prevalent when dealing with various types of heterogeneous data. We introduce a systematic, generalized method, called iSOM-GSN, used to transform ``multi-omic'' data with higher dimensions onto a two-dimensional grid. Afterwards, we apply a convolutional neural network to predict disease states of various types. Based on the idea of Kohonen's self-organizing map, we generate a two-dimensional grid for each sample for a given set of genes that represent a gene similarity network. We have tested the model to predict breast and prostate cancer using gene expression, DNA methylation and copy number alteration, yielding prediction accuracies in the 94-98% range for tumor stages of breast cancer and calculated Gleason scores of prostate cancer with just 11 input genes for both cases. The scheme not only outputs nearly perfect classification accuracy, but also provides an enhanced scheme for representation learning, visualization, dimensionality reduction, and interpretation of the . Large scale projects such as "The Cancer Genome Atlas" (TCGA) generate a plethora of multidimensional data by applying high-resolution microarrays and next generation sequencing. This leads to diverse multi-dimensional data in which the need for devising dimensionality reduction and representation learning methods to integrate and analyze such data arises. An earlier study by Shen et al. proposed algorithms iCluster (a) and iCluster+ (b), which made use of the latent variable model and principal component analysis (PCA) on multi-omic data and aimed to cluster cancer data into sub-types; even though it performed well, it did not use multi-omics data. In another study, attempted to apply heatmaps as a dimensionality reduction scheme on gene expression data to deduce biological insights and then classify cancer types from a Pan-cancer cohort. However, the accuracy obtained by using that method was limited to 97% on Pan-cancer data, lacking the benefits of integrated multi-omics data. In a recent study used self-Organizing maps (SOMs) to embed gene expression data into a lower dimensional map, while the works of (; ; ;) generate clusters using SOMs on gene expression data with different aims. In addition, the work of combines gene expression and DNA methylation to identify subtypes of cancer similar to those of , which identifies modules of co-expressing genes. On the other hand, the work of uses SOMs to create a generalized regression neural network, while the model proposed in uses SOMs to classify documents based on a word-tovector model. Apart from dimensionality reduction methods, attempts have been made by applying supervised deep machine learning, such as deepDriver , which predicts candidate driver genes based on mutation-based features and gene similarity networks. Although these works have been devised to use embedding and conventional machine learning approaches, the use deep neural networks on multi-omics data integration is still in its infancy. In addition, these methods lack Gleason Score Number of Samples Group 3+4 147 34 4+3 101 43 4+5,5+4 139 9 Table 1: Distribution of the different Gleason groups considered for PRCA. in adequacy to generalize them multi-omics data to predic disease states. More specifically, none of these models combine the strength of SOMs for representation learning combined with the CNN for image classification as we do in this work. In this paper, a deep learning-based method is proposed, and is used to predict disease states by integrating multi-omic data. The method, which we call iSOM-GSN, leverages the power of SOMs to transform multi-omic data into a gene similarity network (GSN) by the use of gene expression data. Such data is then combined with other genomic features to improve prediction accuracy and help visualization. To our knowledge, this the first deep learning model that uses SOMs to transform multi-omic data into a GSN for representation learning, and uses CNNs for classification of disease states or other clinical features. The main contributions of this work can be summarized as follows: • A deep learning method for prediction of tumor aggressiveness and progression using iSOM-GSN. • A new strategy to derive gene similarity networks via self-organizing maps. • Use of iSOM-GSN to identify relevant biomarkers without handcrafted feature engineering. • An enhanced scheme to interpret and visualize multi-dimensional, multi-omics data. • An efficient model for graph representation learning. We considered two datasets as part of our study: The Cancer Genome Atlas (TCGA) Prostate Adenocarcinoma (PRCA) and The Cancer Genome Atlas (TCGA) Breast Invasive Carcinoma (BRCA) . Our aim here is to classify patients based on Gleason scores for PRCA , and tumor stage for BRCA . The total number of samples for PRCA and BRCA were 499 and 570 respectively. Both datasets had approximately 60,000 features for gene expression data alone. Thus, a variance threshold of 0.2% was applied to these data, which removes all features that have at least 80% zero values; this step reduced the feature set size to 16,000. The data were then normalized on a common scale for all omics, including DNA methylation and CNA data. The gene names were preserved in HUGO format and the names considered irrelevant by HUGO were removed. All the three types of data were then combined, based on patient ID, which yielded data for 387 and 392 patients for PRCA and BRCA respectively, containing all three required omic data. Since imbalance was observed across all classes in the PRCA dataset, we considered only three distinct Gleason scores. It is worthwhile to note that samples with Gleason score 7 were considered as two different classes, i.e., 3+4 and 4+3, for example, since these two groups are clinically different. More details on class distribution are shown in Tables 1 and 2. MultisgCV was used to further process the data . The MutsigCV algorithm identifies significantly mutated genes by building a patient-specific mutation model based on gene expression and DNA methylation data. This method takes the whole genome or exome sequence as input and identifies genes that are mutated more often. The top 14 mutated genes from Multisig were considered for the rest of the experiment. 2A 179 2B 129 3A 84 Table 2: Distribution of the different tumor groups considered for BRCA. We consider the problem of integrating multiple types of omics data. For this purpose, we propose a three-step approach, which we call iSOM-GSN, and whose main steps are depicted in Figure 1. First, we create a GSN by extracting features from one data type, in our case, gene expression data. Then, for each sample, we integrate all data types by considering features extracted from the first step. Finally, we apply a CNN to perform classification with training and test split at 70:30 ratio to test the model. We assume the input data is a set of matrices S = {s oj}, where i = {1, 2, 3, . . ., n} represents the samples, j = {1, 2, 3, . . ., m} represents the genes, and o = {1, 2, 3, . . ., p} represents types of data (omics). Here, n is the number of samples, m is the number of genes, and p is the number of types of omics. The first step consists of creating a gene similarity network (GSN) by applying a self-organizing map learning algorithm. In this step, we consider only one type of data, i.e., gene expression. Let i,j=1 denote one omic data where j = {1, 2, 3, . . ., m} represents the set of genes and i = {1, 2, 3, . . ., n} represents the set of samples. S 1 is the input to the SOM. A SOM is a lower-dimensional representation of complex, higher-dimensional data in such a way that distances among vectors in the original space are preserved in the new representation. A SOM is learned via an unsupervised clustering algorithm, which takes sample vectors as inputs, and groups them based on the similarities derived by the features. In our case, the input vectors to the SOM are the samples with gene expression values of all samples as features. The following are the main steps that are followed to construct a SOM. 1. Initialize m neurons with random weights assigned to each neuron c k, where k = 1, 2,..., m, where m is the number of genes under consideration, in our case 14. 2. Calculate the Euclidean distance between each gene g j and its neuron c k, and identify the winning neuron, i.e., the neuron that has the smallest distance to its respective neuron. The Euclidean distance is calculated as follows: where s 1j = g j represents the gene vector for i th sample and c j1 represents neuron vector. 3. Suppose that c k is the winning neuron, i.e., it is the closest to gene g j. Then, we update the weight of c k using Equation. The winning neuron is also known as best matching unit (BMU). 4. Update the weights of the neurons that are in proximity to the BMU, c k. To account for this, we use a neighbourhood function that is defined by Equation. 5. Repeat steps 2 -4 for e iterations or until desired convergence (i.e., the weights remain unchanged or the change is less than a threshold). 6. Finally, obtain c m neurons, which represent g m genes in the two-dimensional space, represented by Equation. where L(t) is the learning rate regulation function defined in Equation. where L 0 is initial learning rate. where (x j, y j) represents the coordinates of g j. As a of running the training algorithm, a SOM is obtained in which the genes are organized based on their similarity, representing a GSN. This network is represented as a two-dimensional lattice whose coordinates are denoted as in Equation. Figure 2 shows the two SOMs derived from the two datasets, BRCA and PRCA. Observing the evolution of the SOM learning algorithms through the different epochs for both datasets (see Figures 10-13 of the Supplementary Material), show how complex, high-dimensional relationships among related genes are revealed and visualized in a simple way on a two-dimensional map. The second step of iSOM-GSN is to integrate multiple data types. We use the GSN generated in the first step as a template image; in the example, the genes are indexed with numbers by followng the mapping listed in Table 3. We then expand a circular region around the points with a predefined radius and color the circles as shown in Figure 3. We color each circle by considering each data view by using the RGB color scheme, where Red is represented by gene expression, Green by DNA methylation and Blue by CNA. In our case, S Index Gene Name BRCA Gene Name PRCA 0 RUNX1 SPOP 1 PIK3CA FOXA1 2 GATA3 CTNNB1 3 FOXA1 CLPTM1L 4 SF3B1 DPYSL2 5 PTEN NEIL1 6 CBFB PITPNM2 7 CDH1 ATM 8 MAP2K4 EMG1 9 MAP3K1 ETV3 10 ERBB2 BRAF 11 NCOR1 NKX3-1 12 FAM86B2 ZMYM3 13 CDKN1B SALL1 Table 3: Indices of gene names for the BRCA and PRCA datasets. If point (p, q) is within certain radius of g j, 0 otherwise. where and B As a , we obtain a set of matrices, one per each sample, defined as follows: Figure 3 represents a sample image created after integrating multiple omics for the BRCA dataset. As can be observed, various shades of colors for different genes represent their values with respect to the three different types of omic data. The last step of iSOM-GSN is to feed the images generated in the previous step to the CNN, to predict the state of the disease as the final output. The architecture of the CNN is proven to be the most effective in learning visual representations. The CNN is also known to perform better than the human eye in many visual processing problems. The usage of the CNN in any method is just a variation in how the convolution and pooling layers are combined, and how the network is trained. A more detailed, schematic diagram of the entire network design is depicted in Figure 7 of the Supplementary Material. The network includes two convolutional layers and two fully-connected layers with a small number of neurons. Our choice of a smaller network design is motivated both from our desire to reduce the risk of over-fitting as well as to simplify the nature of the classification. All three color channels, i.e., RGB, are processed directly by the network. The subsequent convolutional and fully connected layers are then defined as follows: • 32 filters of size 3 × 3 pixels are applied to the input in the first convolutional layer, followed by a rectified linear operator (ReLU), a Max-pooling layer taking the maximal value of 2 × 2 regions with two-pixel strides and a local response normalization layer. • The output of the previous layer is then processed by the second convolutional layer, containing 32 filters of size 3 × 3 pixels. Again, this is followed by ReLU, a Max-pooling layer and a local response normalization layer with the same hyper-parameters as before. • First fully connected layer that receives the output of the second convolutional layer and contains 128 neurons, followed by ReLU and a dropout layer. • Second fully connected layer that receives the output of the first fully connected layer and output three neurons, followed by ReLU and a dropout layer. Finally, the output of the last fully-connected layer is fed to a Soft-max layer that assigns a probability to each class. The prediction itself is performed by applying Soft-max to choose the class with maximal probability for the given test image. Aside from using a lean network architecture, i.e., fewer layers, we apply two additional methods to further limit the risk of over-fitting. First, we apply dropout learning, i.e., randomly setting the output value of the network neurons to zero. The network includes three dropout layers with a dropout ratio of 0.5 (50% chance of setting a neuron's output value to zero). Second, we use data augmentation by taking a random input image, and scaling and mirroring it in each forward-backward training pass. Training is done using the Adam optimizer . To assess the performance of iSOM-GSN, the data was divided into training and test datasets with a ratio 70:30. Minmax scaling was then applied on the test dataset followed by ranging the training dataset accordingly. Note that the test data is scaled using the same criterion applied to the training dataset, and without any information about the classes. We then calculated the main performance measures that include categorical accuracy, precision, recall, F1-score and mean absolute error. iSOM-GSN has been run on two multi-omic datasets namely PRCA and BRCA using the model and parameters described earlier in this paper. Figure 4 depicts the plot of how various performance measures convolve with an increasing number of epochs. In general, it can be seen that the predictive performance is almost perfect with respect to various parameters. However, regarding the number of genes obtained after filtering, both datasets used the exact number of genes retained for effective classification. In addition, Figure 6 of the Supplementary Material depicts the plot of the receiver operating characteristic (ROC), area under curve (AUC). In general terms, it can be seen that the predictive performance is in the range 94-98% with respect to various parameters. However, regarding the number of genes obtained after filtering, both datasets used the same number of genes retained for effective classification. This shows that only 14 genes are enough to classify any clinical variable using the proposed model, and those genes are significantly mutated. We illustrate the ability to discover and visualize patterns of genomic interactions in biological comprehensive context for classification. When the goal is to identify potential biomarkers and factors that characterize biological and clinical aspects, the proposed approach comes to the rescue. As part of a feature selection step, we narrowed down the relevant features to just 14 genes, which are sufficient for classification and achieve 96% accuracy. These genes are listed in Table 5. All genes are identified either as tumor suppressor genes or oncogenes for known pathways as shown in Table 5. TP53 and PTEN are the common genes that are highly mutated in both PRCA and BRCA datasets. These gene are tumor suppressor genes, which are well known to express high gene expression values and are considered as biomarkers for cancer in general (https://www.genecards.org/, 2019). On the other hand, SALL1 and PITPNM2 are genes that are not known as cancer related genes. While validating the genes related to BRCA data set, we have found in that genes MAP3K1 and MAP2K4 are strong predictors of MEK inhibitors which are frequently found in breast, prostate and colon cancers. These can be potential targets for drugs . In another study, we have found co-occurring mutations of PIK3CA and MAP3K1 are functionally significant in breast cancer and MAP3K1 mutational status may be considered as a predictive biomarker for efficacy in PI3K pathway inhibitor trials . We have also found that expression of genes GATA-3 and FOXA1 are sufficient to differentiate breast carcinoma from other and hence are excellent bio-markers . This is also supported by the findings reported in Figure 5: Most relevant genes found to predict Gleason groups of PRCA and stages of BRCA. , which claim that these two genes are associated with a less aggressive phenotype and they give better prognosis in patients with HR-positive or HER2-negative breast cancer. Thus, we confirm that the GSN's formed are helpful to find potential novel bio-markers. A practical challenge for validating this framework is the unavailability of independent datasets with all data types. An intrinsic question arises as to what extent a single data type (e.g., gene expression) is effective in classifying, based on our framework. The closest method to compare our work with is the one that uses gene expression and DNA methylation , though they use the two types of data separately, and apply it to a single disease. To that end, we ran iSOM-GSN on a single omic data at a time. We discovered that gene expression data alone yielded 90% accuracy, whereas DNA methylation and CNA, alone, yielded 87% and 89% classification accuracy, respectively. This demonstrates the advantages of combining multi-omics, the strengths of the SOMs for representation learning combined with the power of deep CNNs for image-based data classification. This paper presents a framework that uses a self-organizing map and a convolutional neural network used to conduct data integration, representation learning, dimensionality reduction, feature selection and classification simultaneously to harness the full potential of integrated high-dimensional large scale cancer genomic data. We have introduced a new way to create gene similarity networks, which can lead to novel gene interactions. We have also provided a scheme to visualize high-dimensional, multi-omics data onto a two-dimensional grid. In addition, we have devised an approach that could also be used to integrate other types of multi-omic data and predict any clinical aspects or states of diseases, such as laterality of the tumor, survivability, or cancer sub types, just to mention a few. This work can also be extended to classify Pan-cancer data. Omics can be considered as a vector and more than three types of data (i.e., beyond RGB images) can be incorporated for classification. Apart from integrating multi-omics data, the proposed approach can be considered as an unsupervised clustering algorithm, because of the competitive learning nature of SOMs. We can also apply iSOM-GSN on other domains, such as predicting music genre's for users based on their music preference. As a first step, we have applied the SOM to a Deezer dataset and the are encouraging 14. Applications of iSOM-GSN can also be in drug response or re-purposing, prediction of passenger or oncogenes, revealing topics in citation networks, and other prediction tasks. The source code has been posted on a Github project, available at the following anonymous website: https://gitlab.com/NF2610/isom_gsn. Figure 6: ROC plots for the proposed model run on the PRCA dataset.

represent the tldr for finding the corresponding scientific article
Given A GAN based method to learn important topological features of an arbitrary input graph., a positive would be Inspired by the success of generative adversarial networks (GANs) in image domains, we introduce a novel hierarchical architecture for learning characteristic topological features from a single arbitrary input graph via GANs. The hierarchical architecture consisting of multiple GANs preserves both local and global topological features, and automatically partitions the input graph into representative stages for feature learning. The stages facilitate reconstruction and can be used as indicators of the importance of the associated topological structures. Experiments show that our method produces subgraphs retaining a wide range of topological features, even in early reconstruction stages. This paper contains original research on combining the use of GANs and graph topological analysis. Graphs have great versatility, able to represent complex systems with diverse relationships between objects and data. With the rise of social networking, and the importance of relational properties to the "big data" phenomenon, it has become increasingly important to develop ways to automatically identify key structures present in graph data. Identification of such structures is crucial in understanding how a social network forms, or in making predictions about future network behavior. To this end, a large number of graph analysis methods have been proposed to analyze network topology at the node BID57, community BID41 BID55, and global levels BID64.Each level of analysis is greatly influenced by network topology, and thus far algorithms cannot be adapted to work effectively for arbitrary network structures. Modularity-based community detection BID65 works well for networks with separate clusters, whereas edge-based methods BID37 are suited to dense networks. Similarly, when performing graph sampling, Random Walk (RW) is suitable for sampling paths BID51, whereas Forrest Fire (FF) is useful for sampling clusters BID52. When it comes to graph generation, Watts-Strogatz (WS) graph models BID62 can generate graphs with small world features, whereas Barabsi-Albert (BA) graph models BID31 simulate super hubs and regular nodes according to the scale-free features of the network. However, real-world networks typically have multiple topological features. Considering real-world networks also introduces another issue that traditional graph analysis methods struggle with; having a mere single instance of a graph (e.g. the transaction graph for a particular bank), making it difficult to identify the key topological properties in the first place. In particular, we are interested in both "local topological features" (such as the presence of subgraph structures like triangles) and "global topological features" such as degree distribution. Instead of directly analyzing the entire topology of a graph, GTI first divides the graph into several hierarchical layers. A hierarchical view of a graph can split the graph by local and global topological features, leading to a better understanding of the graph BID63. As different layers have different topological features, GTI uses separate GANs to learn each layer and the associated features. By leveraging GANs renowned feature identification BID42 on each layer, GTI has the ability to automatically capture arbitrary topological features from a single input graph. Figure 1 demonstrates how GTI can learn to reproduce an input graph where a single GAN cannot. In addition to learning topological features from the input graph, the GTI method defines a reconstruction process for reproducing the original graph via a series of reconstruction stages (the number Figure 1 : How GTI recovers the original graph while naive GAN methods do not: The DCGAN output looks like a complete graph, whereas GTI can capture the super-hub structure of node 3 and node 2.of which is automatically learned during training). As stages are ranked in order of their contribution to the full topology of the original graph, early stages can be used as an indicator of the most important topological features. Our focus in this initial work is on the method itself and demonstrating our ability to learn these important features quickly (via demonstrating the retention of identifiable structures and comparisons to graph sampling methods). In this section, we demonstrate the work flow of GTI (see Figure 2), with a particular focus on the GAN, Sum-up and Stage Identification modules. At a high level, the GTI method takes an input graph, learns its hierarchical layers, trains a separate GAN on each layer, and autonomously combines their output to reconstruct stages of the graph. Here we give a brief overview of each module. Hierarchical Identification Module: This module detects the hierarchical structure of the original graph using the Louvain hierarchical community detection method BID33, denoting the number of layers as L. The number of communities in each layer is used as a criterion for how many subgraphs a layer should pass to the next module. Layer Partition Module: Here we partition a given layer into M non-overlapping subgraphs, where M is the number of communities. We do not use the learned communities from the Louvain method as we cannot constrain the size of any community. We instead balance the communities into fixed size subgraphs using the METIS approach BID46.Layer GAN Module: Rather than directly using one GAN to learn the whole graph, we use different GANs to learn features for each layer separately. If we use a single GAN to learn features for the whole graph, some topological features may be diluted or even ignored. This module regenerates subgraphs, with the Layer Regenerate Module generating the adjacency matrix for the corresponding layer. For more detail see Section 2.1.Layer Regenerate Module: Here, for a given layer, the corresponding GAN has learned all the properties of each subgraph, meaning we can use the generator in this GAN to regenerate the topology of the layer by generating M subgraphs of k nodes. Note that this reconstruction only restores edges within each non-overlapping subgraph, and does not include edges between subgraphs. All Layer Sum-up Module: This module outputs a weighted reconstructed graph by summing up all reconstructed layers along with the edges between subgraphs that were not considered in the Regenerate Module. The "weight" of each edge in this module represents its importance to the reconstruction. Indeed, we rely upon these weights to identify the reconstruction stages for the original graph. For more details, see Section 2.2.Stage Identification Module: By analyzing the weighted adjacency matrix of the Sum-up Module, we extract stages for the graph. These stages can be interpreted as steps for a graph reconstruction process. See Section 2.3 for details. Where the generator is a deconvolutional neural network with the purpose of restoring a k × k adjacency matrix from the standard uniform distribution, the discriminator is instead a CNN whose purpose is to estimate if the input adjacency matrix is from a real dataset or from a generator. Here, BN is short for batch normalization which is used instead of max pooling because max pooling selects the maximum value in the feature map and ignores other values, whereas BN will synthesize all available information. LR stands for the leaky ReLU active function (LR = max(x, 0.2 × x)) which we use since 0 has a specific meaning for adjacency matrices. In addition, k represents the size of a subgraph, and F C the length of a fully connected layer. We set the stride for the convolutional/deconvolutional layers to be 2. We adopt the same loss function and optimization strategy (1000 iterations of ADAM BID47) with a learning rate of 0.0002) used in the DCGAN method of BID60. ) to add the graphs from all layers together. re G is the reconstructed adjacency matrix (with input from all layers), G i, i ∈ L is the reconstructed adjacency matrix for each layer (with G representing the full original graph with N nodes), E refers to all the inter-subgraph (community) edges identified by the Louvain method from each hierarchy, and b represents a bias. While each layer of the reconstruction may lose certain edge information, summing up the hierarchical layers along with E will have the ability to reconstruct the entire graph. DISPLAYFORM0 To obtain w and b for each layer, we use Equation 2 as the loss function (where we add = 10 DISPLAYFORM1 to avoid taking log or division by 0), minimizing over 500 iterations of SGD with learning rate 0.1. We note that Equation 2 is similar to a KL divergence, though of course re G and G are not probability distributions. DISPLAYFORM2 re G can be interpreted as representing how each edge contributes to the entire topology. According to these weights we can then divide the network into several stages, with each stage representing a collection of edges greater than a certain weight. We introduce the concept of a "cut-value" to turn re G into a binary adjacency matrix. We observe that many edges in re G share the same weight, which implies these edges share the same importance. Furthermore, the number of unique weights can define different reconstruction stages, with the most important set of edges sharing the highest weight. Each stage will include edges with weights greater than or equal to the corresponding weight of that stage. Hence, we define an ordering of stages by decreasing weight, giving insight on how to reconstruct the original graph in terms of edge importance. We denote the ith largest unique weight-value as CV i (for "cut value") and thereby define the stages as in Equation 3 (an element-wise product), where I[w ≥ CV i] is an indicator function for each weight being equal or larger than the CV i. DISPLAYFORM3 In Section 4, we use synthetic and real networks to show that each stage preserves identifiable topological features of the original graph during the graph reconstruction process. As each stage contains a subset of the original graphs edges, we can interpret each stage as a sub-sampling of the original graph. This allows us to compare with prominent graph sampling methodologies to emphasize our ability to retain important topological features. The importance of deep learning and the growing maturity of graph topology analysis has led to more focus on the ability to use the former for the latter BID54. A number of supervised and semi-supervised learning methods have been developed for graph analysis. A particular focus is on the use of CNNs BID34 BID44 BID39 BID36. These new methods have shown promising for their respective tasks in comparison to traditional graph analysis methods (such as kernel-based methods, graph-based regularization techniques, etc).Since GANs were first introduced BID42, its theory and application has expanded greatly. Many advances in training methods BID38 BID67 BID59 have been proposed in recent years, and this has facilitated their use in a number of applications. For example, GANs have been used for artwork synthesis ), text classification BID56, image-to-image translation BID66, imitation of driver behavior BID49, identification of cancers BID48, and more. The GTI method expands the use of GANs into the graph topology analysis area. A distinguishing feature of our method is that it is an unsupervised learning tool (facilitated by the use of GANs) that leverages the hierarchical structure of a graph. GTI can automatically capture both local and global topological features of a network. To the best of the authors' knowledge, this is the first such unsupervised method. All experiments in this paper were conducted locally on CPU using a Mac Book Pro with an Intel Core i7 2.5GHz processor and 16GB of 1600MHz RAM. Though this limits the size of our experiments in this preliminary work, the extensive GAN literature (see Section 3) and the ability to parallelize GAN training based on hierarchical layers suggests that our method can be efficiently scaled to much larger systems. We use a combination of synthetic and real datasets. Through the use of synthetic datasets with particular topological properties, we are able to demonstrate the retention of these easily identifiable 62, 3.87, 26.64, 27.98, 31.79, 32.42, 34.22, 34.65, 34.80, 64.06, 76.81, 100 P2P 3334 6627 7 49.04, 53.90, 70.32, 87.54, 88.40, 89.65, 100 properties across the reconstruction stages. Of course, in real-world applications we do not know the important topological structures a priori, and so also demonstrate our method on a number of real-world datasets of varying sizes. We use the ER graph model BID40, the BA graph model BID31, the WS graph model BID63, and the Kronecker graph model BID53 to generate our synthetic graphs. The varying sizes of our synthetic graphs (as well as our real-world datasets) are outlined in TAB0. The ER (p = 0.2), WS (k = 2, p = 0.1) and BA (m = 2) graphs were generated using the NetworkX package BID43. The Kronecker graph was generated using the krongen package of the SNAP project 1.For real datasets, we use data available from the Stanford Network Analysis Project BID50. In particular, we use the Facebook network, the wiki-Vote network, and the P2P-Gnutella network. The Facebook dataset consists of "friends lists", collected from survey participants according to the connections between user-accounts on the online social network. It includes node features, circles, and ego networks; all of which has been anonymized by replacing the Facebookinternal ids. Wiki-vote is a voting network (who votes for whom etc) that is used by Wikipedia to elect page administrators; P2P-Gnutella is a peer-to-peer file-sharing network: Nodes represent hosts in the Gnutella network topology, with edges representing connections between the hosts. RoadNet is a connected component of the road network of Pennsylvania. Intersections and endpoints are represented by nodes, and the roads connecting them are edges. Here we use two examples to demonstrate how GTI retains important local topological structure during each reconstruction stage. We demonstrate the reconstruction process of a BA network in Figure 4, with the top row demonstrating the entire reconstruction process of the full network. We clearly observe that each reconstructed network becomes denser and denser as additional stages are added. The bottom row of Figure 4 shows the subgraphs corresponding to nodes 0 to 19 at each reconstruction stage. We observe that these subgraphs retain the most important feature of the original subgraph (the star structures at node 0), even during the first reconstruction stage. Road Network Stages Analysis: We observe in TAB0 that the retained edge percentages of the RoadNet reconstruction decrease more consistently with each stage than in the BA network. This is reasonable, because geographical distance constraints naturally in fewer super hubs, with each node having less variability in its degree. In Figure 5, we observe the reconstruction of the full network, and the node 0 to node 19 subgraph of RoadNet. We observe in the bottom row of Figure 5 that the dominant cycle structure of the original node 0-19 subgraph clearly emerges. We also observe an interesting property of the stages of the original graph in the top row of Figure 5. As SNAP does not provide the latitude and longitude of nodes, we cannot use physical location. We instead calculate the modularity of each stage, where modularity represents the tightness of the community BID58. We found that the modularity deceases from 0.98 to 0.92 approximately linearly. This indicates that GTI views the dense connections between local neighborhoods as a particularly representative topological property of road networks, as such clusters are formed before links between clusters. In the previous section, we demonstrated GTI's ability to the preserve local topological features. Here we focus on degree distribution and the distribution of cluster coefficients, global topological features. Figure 6 and Figure 7 respectively show the log-log degree distributions and log-log cluster coefficient distributions for each of the datasets given in TAB0, where the horizontal axis represents the ordered degrees (ordered cluster coefficient), and the vertical axis the corresponding density. The red line is used to demonstrate the degree distribution (cluster coefficient distribution) of the original graph, with the distributions of the ordered stages represented by a color gradient from green to blue. We observe that with the exception of the ER network, the degree distributions and the cluster coefficient distributions of early stages are similar to the original graphs, and only become more accurate as we progress through the reconstruction stages. Although the degree distributions and cluster coefficient distributions for the early stages of the ER network reconstruction are shifted, we observe that GTI quickly learns the Poisson like shape in degree distribution, and also learns the "peak-like" shape in the cluster coefficient distribution. This is particularly noteworthy given that the ER model has no true underlying structure (as graphs are chosen uniformly at random). Finally, we note that GTI quickly learns that the cluster coefficient of the WS network is zero. The graphs generated by GTI can be considered as samples of the original graph in the sense that they are representative subgraphs of a large input graph. We compare the performance of GTI with that of other widely used graph sampling algorithms (Random Walk, Forest Fire and Random Jump) with respect to the ability to retain topological structures BID51.We demonstrate this through the subgraph structures of the BA and Facebook datasets, comparing stage 1 of GTI against the graph sampling algorithms (designed to terminate with the same number Log of Degrees Log of Density of nodes as the GTI stage). We take out each of subgraphs from BA and Facebook network (nodes 0-19 and nodes 0-49) to visually compare the ability of the first stage of GTI to retain topological features in comparison to the three graph sampling methods. In Figure 8, we observe that stage 1 of GTI has retained a similar amount of structure in the 20 node BA subgraph as Forest Fire BID52, while demonstrating considerably better retention than either Random Walk or Random Jump. However, for 50 node BA subgraph, only GTI has the ability to retain the two super hubs present in the original graph. In Figure 9, we observe that GTI demonstrates vastly superior performance to the other methods when run on the Facebook dataset, which has a number of highly dense clusters with very sparse inter-cluster connections. DISPLAYFORM0 DISPLAYFORM1 Original Graph GTI Random Walk represents the node-node similarity with mismatch penalty BID45 between node i ∈ G L−1 and node j ∈ G. Here, the average node-node similarity indicates how G L−1 resembles G.For comparison, we use the same model parameters as before to generate 100 examples of BA, ER, Kronecker, and WS networks. These newly generated networks serve as base models to help us evaluate the similarity between the penultimate stage of GTI and the original graph, as well as the effectiveness of the sampling methods. Unlike the sampling methods, the of TAB1 demonstrate a consistent ability to retain topological features across a variety of different graph types. This paper leveraged the success of GANs in (unsupervised) image generation to tackle a fundamental challenge in graph topology analysis: a model-agnostic approach for learning graph topological features. By using a GAN for each hierarchical layer of the graph, our method allowed us to reconstruct diverse input graphs very well, as well as preserving both local and global topological features when generating similar (but smaller) graphs. In addition, our method identifies important features through the definition of the reconstruction stages. A clear direction of future research is in extending the model-agnostic approach to allow the input graph to be directed and weighted, and with edge attributes.

This paper presents a deep learning model that combines self-organizing maps and convolutional neural networks for representation learning of multi-omics data

represent the tldr for finding the corresponding scientific article.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.