Question: represent the article for finding a single-sentence tldr

Question: Most existing deep reinforcement learning (DRL) frameworks consider action spaces that are either discrete or continuous space. Motivated by the project of design Game AI for King of Glory (KOG), one the world’s most popular mobile game, we consider the scenario with the discrete-continuous hybrid action space. To directly apply existing DLR frameworks, existing approaches either approximate the hybrid space by a discrete set or relaxing it into a continuous set, which is usually less efficient and robust. In this paper, we propose a parametrized deep Q-network (P-DQN) for the hybrid action space without approximation or relaxation. Our algorithm combines DQN and DDPG and can be viewed as an extension of the DQN to hybrid actions. The empirical study on the game KOG validates the efficiency and effectiveness of our method. In recent years, the exciting field of deep reinforcement learning (DRL) have witnessed striking empirical achievements in complicated sequential decision making problems that are once believed unsolvable. One active area of the application of DRL methods is to design artificial intelligence (AI) for games. The success of DRL in the game of Go provides a promising methodology for game AI. In addition to the game of Go, DRL has been widely used in other games such as Atari BID19, Robot Soccer BID8 BID17, and Torcs ) to achieve super-human performances. However, most existing DRL methods only handle the environments with actions chosen from a set which is either finite and discrete (e.g., Go and Atari) or continuous (e.g. MuJoCo and Torcs) For example, the algorithms for discrete action space include deep Q-network (DQN) BID18, Double DQN , A3C BID20; the algorithms for continuous action space include deterministic policy gradients (DPG) BID29 and its deep version DDPG.Motivated by the applications in Real Time Strategic (RTS) games, we consider the reinforcement learning problem with a discrete-continuous hybrid action space. Different from completely discrete or continuous actions that are widely studied in the existing literature, in our setting, the action is defined by the following hierarchical structure. We first choose a high level action k from a discrete set {1, 2, · · ·, K}; upon choosing k, we further choose a low level parameter x k ∈ X k which is associated with the k-th high level action. Here X k is a continuous set for all k ∈ {1, . . ., K}.1 Therefore, we focus on a discrete-continuous hybrid action space A = (k, x k) x k ∈ X k for all 1 ≤ k ≤ K.To apply existing DRL approaches on this hybrid action space, two straightforward ideas include:• Approximate A by an finite discrete set. We could approximate each X k by a discrete subset, which, however, might lose the natural structure of X k. Moreover, when X k is a region in the Euclidean space, establishing a good approximation usually requires a huge number discrete actions.• Relax A into a continuous set. To apply existing DRL framework with continuous action spaces, BID8 define the following approximate space DISPLAYFORM0 where F k ⊆ R. Here f 1, f 2,..., f K is used to select the discrete action either deterministically (by picking arg max i f i) or randomly (with probability softmax(f)). Compared with the original action space A, A might significantly increases the complexity of the action space. Furthermore, continuous relaxation can also lead to unnecessary confusion by over-parametrization. For example, (1, 0, · · ·, 0, x 1, x 2, x 3, · · ·, x K) ∈ A and (1, 0, · · ·, 0, x 1, x 2, x 3, · · ·, x K) ∈ A indeed represent the same action (1, x 1) in the original space A.In this paper, we propose a novel DRL framework, namely parametrized deep Q-network learning (P-DQN), which directly work on the discrete-continuous hybrid action space without approximation or relaxation. Our method can be viewed as an extension of the famous DQN algorithm to hybrid action spaces. Similar to deterministic policy gradient methods, to handle the continuous parameters within actions, we first define a deterministic function which maps the state and each discrete action to its corresponding continuous parameter. Then we define a action-value function which maps the state and finite hybrid actions to real values, where the continuous parameters are obtained from the deterministic function in the first step. With the merits of both DQN and DDPG, we expect our algorithm to find the optimal discrete action as well as avoid exhaustive search over continuous action parameters. To evaluate the empirical performances, we apply our algorithm to King of Glory (KOG), which is one of the most popular online games worldwide, with over 200 million active users per month. KOG is a multi-agent online battle arena (MOBA) game on mobile devices, which requires players to take hybrid actions to interact with other players in real-time. Empirical study indicates that P-DQN is more efficient and robust than BID8's method that relaxes A into a continuous set and applies DDPG. In reinforcement learning, the environment is usually modeled by a Markov decision process (MDP) M = {S, A, p, p 0, γ, r}, where S is the state space, A is the action space, p is the Markov transition probability distribution, p 0 is the probability distribution of the initial state, r(s, a) is the reward function, and γ ∈ is the discount factor. An agent interacts with the MDP sequentially as follows. At the t-th step, suppose the MDP is at state s t ∈ S and the agent selects an action a t ∈ A, then the agent observe an immediate reward r(s t, a t) and the next state s t+1 ∼ p(s t+1 |s t, a t). A stochastic policy π maps each state to a probability distribution over A, that is, π(a|s) is defined as the probability of selecting action a at state s. Whereas a deterministic µ: S → A maps each state to a particular action in A. Let R t = j≥t γ j−t r(s j, a j) be the cumulative discounted reward starting from time-step t. We define the state-value function and the action-value function of policy π as V π = E(R t |S t = s; π) and Q π (s, a) = E(R t |S 0 = s, A 0 = a; π), respectively. Moreover, we define the optimal state-and action-value functions as V π = sup π V π and Q * = sup π Q π, respectively, where the supremum is taken over all possible policies. The goal of the agent is to find a policy the maximizes the expected total discounted reward J(π) = E(R 0 |π), which is can be achieved by estimating Q *. Broadly speaking, reinforcement learning algorithms can be categorized into two classes: value-based methods and policy-based methods. Value-based methods first estimate Q * and then output the greedy policy with respect to that estimate. Whereas policy-based methods directly optimizes J(π) as a functional of π. The Q-learning algorithm BID36 ) is based on the Bellman equation DISPLAYFORM0 which has Q * as the unique solution. In the tabular setting, the algorithm updates the Q-function by iteratively applying the sample counterpart of the Bellman equation DISPLAYFORM1 where α > 0 is the stepsize and s is the next state observed given the current state s and action a. However, when the state space S is so large that it is impossible to store all the states in memory, function approximation for Q * is applied. Deep Q-Networks (DQN) BID19 approximates Q * using a neural network Q(s, a; w) ≈ Q(s, a), where w is the network weights. In the t-th iteration, the DQN updates the parameter using the gradient of the least squares loss function DISPLAYFORM2 In practice, DQN is trained with techniques such as experience replay and asynchronous stochastic gradient descent methods BID20 ) which enjoy great empirical success. In addition to the value-based methods, the policy-based methods directly models the optimal policy. In specific, let π be any policy. We write p t (·|s; π) as the distribution of S t given S 1 = s with actions executed according to policy π. We define the discounted probability distribution ρ π by DISPLAYFORM3 Then the objective of policy-based methods is to find a policy that maximizes the expected reward DISPLAYFORM4 Let π θ be a stochastic policy parametrized by θ ∈ Θ. For example, π θ could be a neural network in which the last layer is a softmax layer with |A| neurons. The stochastic gradient methods aims at finding a parameter θ that maximizes J(π θ) via gradient descent. The stochastic policy gradient theorem BID33 states that DISPLAYFORM5 3)The policy gradient algorithm iteratively updates θ using estimates of (2.3). For example, the REINFORCE algorithm BID37 updates θ using ∇ θ log π θ (a t |s t) · r t. Moreover, the actor-critic methods use another neural network Q(s, a; w) to estimate the value function Q π θ (s, a) associated to policy π θ. This algorithm combines the value-based and policy-based perspectives together, and is recently used to achieve superhuman performance in the game of Go BID31. When the action space is continuous, value-based methods will no longer be computationally tractable because of taking maximum over the action space A in (2.2), which in general cannot be computed efficiently. The reason is that the neural network Q(s, a; w) is nonconvex when viewed as a function of a; max a∈A Q(s, a; w) is the global minima of a nonconvex function, which is NP-hard to obtain in the worst case. To resolve this issue, the continuous Q-learning BID6 rewrite the action value function as Q(s, a) = V (s) + A(s, a), where V (s) is the state value function and A(s, a) is the advantage function that encodes the relative advantage of each action. These functions are approximated by neural networks V (s; θ V) and A(s, a; θ A), respectively, where θ V and θ A are network weights. The action value function is given by DISPLAYFORM0 Then in the t-th iteration, the continuous Q-learning updates θ v and θ a by taking a gradient step using the least squares loss function DISPLAYFORM1 Moreover, it is also possible to adapt policy-based methods to continuous action spaces by considering deterministic policies. Let µ θ: S → A be a deterministic policy. Similar to (2.3), the deterministic policy gradient (DPG) theorem BID29 states that DISPLAYFORM2 Furthermore, this deterministic version of the policy gradient theorem can be viewed as the limit of (2.3) with the variance of π θ going to zero. Based on (2.4), the DPG algorithm BID29 and the deep deterministic policy gradient (DDPG) algorithm are proposed. General reinforcement learning There is a huge body of literature in reinforcement learning, we refer readers to textbooks by; Szepesvári FORMULA6 for detailed introduction. Combined with the recent advancement of deep learning BID5, deep reinforcement learning becomes a blossoming field of research with a plethora of new algorithms which achieve surprising empirical success in a variety of applications that are previously considered extremely difficult and challenging. Finite discrete action space methods For reinforcement learning problems with finite action spaces, BID18 propose the DQN algorithm, which first combines the deep neural networks with the classical Q-learning algorithm BID36. A variety of extensions are proposed to improve DQN, including Double DQN, dueling DQN BID35, bootstrap DQN BID23, asynchronous DQN BID20, and averaged-.In terms of policy-based methods, BID33 propose the REINFORCE algorithm, which is the basic form of policy gradient. An important extension is the actor-critic method BID13, whose asynchronous deep version A3C BID20 produces the stateof-the-art performances on the Arcade Learning Environment (ALE) benchmark BID1.Continuous action space methods Moreover, for DRL on continuous action spaces, BID29 proposes the deterministic policy gradient algorithm and deterministic actor-critic algorithms. This work is further extended by, which propose the DDPG algorithm, which is an model-free actor critic algorithm using deep neural networks to parametrize the policies. A related line of work is policy optimization methods, which improve the policy gradient method using novel optimization techniques. These methods include natural gradient descent BID12, trust region optimization BID27, proximal gradient descent BID28, mirror descent BID21, and entropy regularization BID22.Hybrid actions A related body of literature is the recent work on reinforcement learning with a structured action space, which contains finite actions each parametrized by a continuous parameter. To handle such parametrized actions, BID8 applies the DDPG algorithm on the relaxed action space directly, and BID17 proposes a learning framework updating the parameters for discrete actions and continuous parameters alternately. Game AI Recently remarkable advances have been made in building AI bots for computer games using deep reinforcement learning. These games include Atari Games, a collection of video games, Texas Hold'em, a multi-player poker game, and Doom, a first-person shooter game. See BID18; BID9; BID15; BID2 for details and see BID11 for a comprehensive survey. More notably, the computer Go agent AlphaGo achieves super-human performances by defeating the human world champion Lee Sedol. Two more complicated class of games are the real-time strategy (RTS) games and MOBA games. These are multi-agent games which involves searching within huge state and action spaces that are possibly continuous. Due to the difficulty of these problems, current research for these games are rather inadequate with most existing work consider specific scenarios instead of the full-fledged RTS or MOBA games. See, e.g., BID4; BID24 for an recent attempt on applying DRL methods to RTS games. This section introduces the proposed framework to handle the application with hybrid discretecontinuous action space. We consider a MDP with a parametrized action space A, which consists of K discrete actions each associated with a continuous parameter. In specific, we assume that any action a ∈ A can be written as a = (k, x k), where k ∈ {1, . . ., K} is the discrete action, and x k ∈ X k is a continuous parameter associated with the k-th discrete action. Thus action a is a hybrid of discrete and continuous components with the value of the continuous action determined after the discrete action is chosen. Then the parametrized action space A can be written as DISPLAYFORM0 In the sequel, we denote {1, . . ., K} by [K] for short. For the action space A in (4.1), we denote the action value function by Q(s, a) = Q(s, k, x k) where s ∈ S, 1 ≤ k ≤ K, and x k ∈ X k. Let k t be the discrete action selected at time t and let x kt be the associated continuous parameter. Then the Bellman equation becomes DISPLAYFORM1 Here inside the conditional expectation on the right-hand side of (4.2), we first solve DISPLAYFORM2, and then take the largest Q(s t+1, k, x * k). Note that taking supremum over continuous space X k is computationally intractable. However, the right-hand side of (4.2) can be evaluated efficiently providing x * k is given. To elaborate this idea, first note that, when the function Q is fixed, for any s ∈ S and k ∈ [K], we can view x Q k (s) = argsup DISPLAYFORM3 as a function of state s. That is, we identify (4.3) as a function x Q k: S → X k. Then we can rewrite the Bellman equation in (4.2) as DISPLAYFORM4 Note that this new Bellman equation resembles the classical Bellman equation in (2.1) with A = [K]. Similar to the deep Q-networks, we use a deep neural network Q(s, k, x k ; ω) to approximate Q(s, k, x k), where ω denotes the network weights. Moreover, for such a Q(s, k, x k ; ω), we approximate x Q k (s) in (4.3) with a deterministic policy network x k (·; θ): S → X k, where θ denotes the network weights of the policy network. That is, when ω is fixed, we want to find θ such that Q s, k, x k (s; θ); ω ≈ sup DISPLAYFORM5 Remark 4.1. Readers who are familiar with the work by BID8, that also claims to handle discrete-continuous hybrid action spaces, may be curious of its difference from the proposed P-DQN. The key differences are as follows.• In BID8, the discrete action types are parametrized as some continuous values, say f. And the discrete action that is actually executed is chosen via k = arg max i f (i). Such a trick actually turns the hybrid action space into a continuous action space, upon which the classical DDPG algorithm can be applied. However, in our framework, the discrete action type is chosen directly by maximizing the action's Q value explicitly.• The Q network in BID8 uses the artificial parameters f as input, which makes it an action-value function estimator of current policy (Q π). While in our framework, the Q network is actually an approximate estimator of the optimal policy's action-value function (Q).• We note that P-DQN is an off-policy method that can use historical data, while it is hard to use historical data in BID8 because there is only discrete action k without parameters f.(a) Network of P-DQN (b) Network of DDPG Figure 1: Illustration of the networks of P-DQN and DDPG BID8. P-DQN selects the discrete action type by maximizing Q values explicitly; while in DDPG, the discrete action with largest f, which can be seen as a continuous parameterization of K discrete action types, is chosen. Also in P-DQN the state and action parameters are feed into the Q-network which outputs K action values for each action type; while in DDPG, the continuous parameterization f, instead of the actual action k taken, is feed into the Q-network. Suppose that θ satisfies (4.4), then similar to DQN, we could estimate ω by minimizing the meansquared Bellman error via gradient descent. In specific, in the t-th step, let ω t and θ t be the weights of the value network and the deterministic policy network, respectively. To incorporate multi-step algorithms, for a fixed n ≥ 1, we define the n-step target y t by DISPLAYFORM0 We define the least squares loss function for ω by DISPLAYFORM1 Moreover, since we aim to find θ that minimizes Q[s, k, x k (s; θ); ω] with ω fixed, we define the loss function for θ by DISPLAYFORM2 Then we update ω t and θ t by gradient-based optimization methods. Moreover, the gradients are given by DISPLAYFORM3 Here ∇ x Q(s, k, x k ; ω) and ∇ ω Q(s, k, x k ; ω) are the gradients of the Q-network with respect to its third argument and fourth argument, respectively. By (5.5) and (5.4) we update the parameters using stochastic gradient methods. In addition, note that in the ideal case, we would minimize the loss function Θ t (θ) in (5.3) when ω t is fixed. From the in stochastic approximation methods BID14, we could approximately achieve such a goal in an online fashion via a two-timescale update rule BID3. In specific, we update ω with a stepsize α t that is asymptotically negligible compared with the stepsize β t for θ. In addition, for the validity of Input: Stepsizes {αt, βt} t≥0, exploration parameter, minibatch size B, the replay memory D, and a probability distribution µ over the action space A for exploration. Initialize network weights ω1 and θ1. for t = 1, 2,..., T doCompute action parameters x k ← x k (s, θt). Select action at = (kt, x k t) according to the -greedy policy at = a sample from distribution µ with probability, (kt, x k t) such that kt = arg max k∈[K] Q(s, k, x k ; ωt) with probability 1 −.Take action at, observe reward rt and the next state st+1. DISPLAYFORM0 Use data {y b, s b, a b} b∈ [B] to compute the stochastic gradient ∇ω Q t (ω) and ∇ θ Θ t (θ) defined in (5.5) and (5.4). Update the parameters by ωt+1 ← ωt − αt · ∇ω Q t (ωt) and θt+1 ← θt − βt · ∇ θ Θ t (θt). end for stochastic approximation, we require {α t, β t} to satisfy the Robbins-Moron condition BID25. We present the P-DQN algorithm with experienced replay in Algorithm 1.Note that this algorithm requires a distribution µ defined on the action space A for exploration. In each step, with probability, the agent sample an random action from µ; otherwise, it takes the greedy action with respect to the current value function. In practice, if each X k is a compact set in the Euclidean space (as in our case), µ could be defined as the uniform distribution over A. In addition, as in the DDPG algorithm , we can also add additive noise to the continuous part of the actions for exploration. Moreover, we use experience replay BID18 to reduce the dependencies among the samples, which can be replaced by more sample-efficient methods such as prioritized replay.Moreover, we note that our P-DQN algorithm can easily incorporate asynchronous gradient descent to speed up the training process. Similar to the asynchronous n-step DQN in BID20, we consider a centralized distributed training framework where each process can compute its local gradient and synchronize with a global parameter server. In specific, each local process runs an independent game environment to generate transition trajectories and use its own transitions to compute gradients with respect to ω and θ. These local gradients are then aggregated across multiple processes to update the global parameters. Note that these local stochastic gradients are independent. Thus tricks such as experience replay can be avoided in the distributed setting. Moreover, aggregating independent stochastic gradient decrease the variance of gradient estimation, which yields better algorithmic stability. We present the asynchronous P-DQN algorithm in Algorithm 2. For simplicity, here we only lay out the algorithm for each local process, which fetches ω and θ from the parameter server and computes the gradient. The parameter server stores the global parameters ω, θ. It updates the global parameters using the gradients sent from the local processes. In addition we use the RMSProp BID10 to update the network parameters, which is shown to be more stable in practice. The game King of Glory is a MOBA game, which is a special form of the RTS game where the players are divided into two opposing teams fighting against each other. Each team has a team base located in either the bottom-left or the top-right corner which are guarded by three towers on each of the three lanes. The towers can attack the enemies when they are within its attack range. Each player controls one hero, which is a powerful unit that is able to move, kill, perform skills, and purchase Input: exploration parameter, a probability distribution µ over the action space A for exploration, the max length of multi step return tmax, and maximum number of iterations Nstep. Initialize global shared parameter ω and θ Set global shared counter Nstep = 0 Initialize local step counter t ← 1. repeat Clear local gradients dω ← 0, dθ ← 0. tstart ← t Synchronize local parameters ω ← ω and θ ← θ from the parameter server. repeat Observe state st and let x k ← x k (st, θ) Select action at = (kt, x k t) according to the -greedy policy at = a sample from distribution µ with probability, (kt, x k t) such that kt = arg max k∈[K] Q(st, k, x k ; ω) with probability 1 −.Take action at, observe reward rt and the next state st+1. t ← t + 1 Nstep ← Nstep + 1 until st is the terminal state or t − tstart = tmax Define the target y = 0 for terminal st DISPLAYFORM0 Update global θ and ω using dθ and dω with RMSProp BID10 ). until Nstep > Nmax equipments. The goal of the heroes is to destroy the base of the opposing team. In addition, for both teams, there are computer-controlled units spawned periodically that march towards the opposing base in all the three lanes. These units can attack the enemies but cannot perform skills or purchase equipments. An illustration of the map is in FIG2, where the blue or red circles on each lane are the towers. During game play, the heroes advance their levels and obtain gold by killing units and destroying the towers. With gold, the heros are able to purchase equipments such as weapons and armors to enhance their power. In addition, by upgrading to the new level, a hero is able to improve its unique skills. Whereas when a hero is killed by the enemy, it will wait for some time to reborn. In this game, each team contains one, three, or five players. The five-versus-five model is the most complicated mode which requires strategic collaboration among the five players. In contrast, the one-versus-one mode, which is called solo, only depends on the player's control of a single hero. In a solo game, only the middle lane is active; both the two players move along the middle lane to fight against each other. The map and a screenshot of a solo game are given in FIG2 -(b) and (c), respectively. In our experiments, we play focus on the solo mode. We emphasize that a typical solo game lasts about 10 to 20 minutes where each player must make instantaneous decisions. Moreover, the players have to make different types of actions including attack, move and purchasing. Thus, as a reinforcement learning problem, it has four main difficulties: first, the state space has huge capacity; second, since there are various kinds of actions, the action space is complicated; third, the reward function is not well defined; and fourth, heuristic search algorithms are not feasible since the game is in real-time. Therefore, although we consider the simplest mode of King of Glory, it is still a challenging game for artificial intelligence. In this section, we applied the P-DQN algorithm to the solo mode of King of Glory. In our experiments, we play against the default AI hero Lu Ban provided by the game, which is a shooter with long attack range. To evaluate the performances, we compared our algorithm with the DDPG algorithm BID8 under fair condition. In our experiment, the state of the game is represented by a 179-dimensional feature vector which is manually constructed using the output from the game engine. These features consist of two parts. The first part is the basic attributes of the two heroes, the computer-controlled units, and buildings such as the towers and the bases of the two teams. For example, the attributes of the heroes include Health Point, Magic Point, Attack Damage, Armor, Magic Power, Physical Penetration/Resistance, and Magic Penetration/Resistance, and the attributes of the towers include Health Point and Attack Damage. The second component of the features is the relative positions of other units and buildings with respect to the hero controlled by the P-DQN player as well as the attacking relations between other units. We note that these features are directly extracted from the game engine without sophisticated feature engineering. We conjecture that the overall performances could be improved with a more careful engineered set of features. We simplify the actions of a hero into K = 6 discrete action types: Move, Attack, UseSkill1, UseSkill2, UseSkill3, and Retreat. Some of the actions may have additional continuous parameters to specify the precise behavior. For example, when the action type is k = Move, the direction of movement is given by the parameter x k = α, where α ∈ [0, 2π]. Recall that each hero's skills are unique. For Lu Ban, the first skill is to throw a grenade at some specified location, the second skill is to launch a missile in a particular direction, and the last skill is to call an airship to fly in a specified direction. A complete list of actions as well as the associated parameters are given in TAB0. The ultimate goal of a solo game is to destroy the opponent's base. However, the final is only available when the game terminates. Using such kind of information as the reward for training might not be very effective, as it is very sparse and delayed. In practice, we manually design the rewards using information from each frame. Specifically, we define a variety of statistics as follows. (In the sequel, we use subscript 0 to represent the attributes of our side and 1 to represent those of the opponent.)• Gold difference GD = Gold 0 − Gold 1. This statistic measures the difference of gold gained from killing hero, soldiers and destroying towers of the opposing team. The gold can be used to buy weapons and armors, which enhance the offending and defending attributes of the hero. Using this value as the reward encourages the hero to gain more gold.• Health Point difference (HPD = HeroRelativeHP 0 − HeroRelativeHP 1): This statistic measures the difference of Health Point of the two competing heroes. A hero with higher Health Point can bear more severe damages while hero with lower Health Point is more likely to be killed. Using this value as the reward encourages the hero to avoid attacks and last longer before being killed by the enemy.• Kill/Death KD = Kills 0 − Kills 1. This statistic measures the historical performance of the two heroes. If a hero is killed multiple times, it is usually considered more likely to lose the game. Using this value as the reward can encourage the hero to kill the opponent and avoid death.• Tower/Base HP difference THP = TowerRelativeHP 0 − TowerRelativeHP 1, BHP = BaseRelativeHP 0 − BaseRelativeHP 1. These two statistics measures the health difference of the towers and bases of the two teams. Incorporating these two statistic in the reward encourages our hero to attack towers of the opposing team and defend its own towers.• Tower Destroyed TD = AliveTower 0 − AliveTower 1. This counts the number of destroyed towers, which rewards the hero when it successfully destroy the opponent's towers.• Winning Game W = AliveBase 0 − AliveBase 1. This value indicates the winning or losing of the game.• Moving forward reward: MF = x + y, where (x, y) is the coordinate of Hero 0: This value is used as part of the reward to guide our hero to move forward and compete actively in the battle field. The overall reward is calculated as a weighted sum of the time differentiated statistics defined above. In specific, the exact formula is r t = 0.5 × 10 −5 (MF t − MF t−1) + 0.001(GD t − GD t−1) + 0.5(HPD t − HPD t−1 DISPLAYFORM0 The coefficients are set roughly inversely proportional to the scale of each statistic. We note that our algorithm is not very sensitive to the change of these coefficients in a reasonable range. In the experiments, we use the default parameters of skills provided by the game environment (usually pointing to the opponent hero's location). We found such kind of simplification does not affect to the overall performance of our agent. In addition, to deal with the periodic problem of the direction of movement, we use (cos(α), sin(α)) to represent the direction and learn a normalized two-dimensional vector instead of a degree (in practice, we add a normalize layer at the end to ensure this). In addition, the 6 discrete actions are not always usable, due to skills level up, lack of Magic Point (MP), or skills Cool Down(CD). In order to deal with this problem, we replace the max k∈ [K] with max k∈[K] and k is usable when selecting the action to perform, and calculating multi-step target as in Equation 5.1.For the network structure, recall that we use a feature vector of 179 dimensions as the state. We set both the value-network and the policy network as multi-layer fully-connected deep networks. The networks are in the same size of 256-128-64 nodes in each hidden layer, with the Relu activation function. During the training and testing processes, we set the frame skipping parameter to 2. This means that we take actions every 3 frames or equivalently, 0.2 second, which adapts to the human reaction time, 0.1 second. We set t max = 20 (4 seconds) to alleviate the delayed reward. In order to encourage exploration, we use -greedy sampling in training with = 0.255. In specific, the first 5 type actions We further smooth the original noisy curves (plotted in light colors) to their running average (plotted in dark colors). In the 3 rows, we plot the average of episode lengths, reward sum averaged for each episode in training, and reward sum averaged for each episode in validation, for the two algorithms respectively. Usually a positive reward sum indicates a winning game, and vice versa. We can see that the proposed algorithm P-DQN learns much faster than its precedent work in our setting. (a) Performance of P-DQN. (b) Performance of DDPG are sampled with probability of 0.05 each and the action "Retreat" with probability 0.005. For actions with additional parameters, since the parameters are in bounded sets, we draw these parameters from a uniform distribution. Moreover, if the sampled action is infeasible, we execute the greedy policy from the feasible ones, so the effective exploration rate is less than. We uses 48 parallel workers with constant learning rate 0.001 in training and 1 worker with deterministic sampling in validation. The training and validating performances are plotted in Figure 3.We implemented the DDPG BID8 ) algorithm within our learning environment to have a fair comparison. The exact network structure is plotted in Figure 1. Each algorithm is allowed to run for 15 million steps, which corresponds to roughly 140 minutes of wall clock time when paralleled with 48 workers. From the experiments , we can see that our algorithm P-DQN can learn the value network and the policy network much faster comparing to the other algorithm. In (a1), we see that the average length of games increases at first, reaches its peak when the two player's strength are close, and decreases when our player can easily defeat the opponent. In addition, in (a2) and (a3), we see that the total rewards in an episode increase consistently in training as well as in test settings. The DDPG algorithm may not be suitable for hybrid actions with both a discrete part and a continuous part. The major difference is that maximization over k when we need to select a action is computed explicitly in P-DQN, instead of approximated implicitly with the policy network as in DDPG. Moreover, with a deterministic policy network, we extend the DQN algorithm to hybrid action spaces of discrete and continuous types, which makes the P-DQN algorithm more suitable for realistic scenarios. Previous deep reinforcement learning algorithms mostly can work with either discrete or continuous action space. In this work, we consider the scenario with discrete-continuous hybrid action space. In contrast of existing approaches of approximating the hybrid space by a discrete set or relaxing it into a continuous set, we propose the parameterized deep Q-network (P-DQN), which extends the classical DQN with deterministic policy for the continuous part of actions. Empirical experiments of training AI for King of Glory, one of the most popular games, demonstrate the efficiency and effectiveness of P-DQN.

Represent the TLDR for finding the corresponding scientific article

A DQN and DDPG hybrid algorithm is proposed to deal with the discrete-continuous hybrid action space.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the article for finding a single-sentence TLDR

Question: Over the past decade, knowledge graphs became popular for capturing structured domain knowledge. Relational learning models enable the prediction of missing links inside knowledge graphs. More specifically, latent distance approaches model the relationships among entities via a distance between latent representations. Translating embedding models (e.g., TransE) are among the most popular latent distance approaches which use one distance function to learn multiple relation patterns. However, they are mostly inefficient in capturing symmetric relations since the representation vector norm for all the symmetric relations becomes equal to zero. They also lose information when learning relations with reflexive patterns since they become symmetric and transitive. We propose the Multiple Distance Embedding model (MDE) that addresses these limitations and a framework which enables collaborative combinations of latent distance-based terms (MDE). Our solution is based on two principles: 1) using limit-based loss instead of margin ranking loss and 2) by learning independent embedding vectors for each of terms we can collectively train and predict using contradicting distance terms. We further demonstrate that MDE allows modeling relations with (anti)symmetry, inversion, and composition patterns. We propose MDE as a neural network model which allows us to map non-linear relations between the embedding vectors and the expected output of the score function. Our empirical show that MDE outperforms the state-of-the-art embedding models on several benchmark datasets. While machine learning methods conventionally model functions given sample inputs and outputs, a subset of Statistical Relational Learning (SRL) approaches specifically aim to model "things" (entities) and relations between them. These methods usually model human knowledge which is structured in the form of multi-relational Knowledge Graphs (KG). KGs allow semantically rich queries and are used in search engines, natural language processing (NLP) and dialog systems. However, they usually miss many of the true relations , therefore, the prediction of missing links/relations in KGs is a crucial challenge for SRL approaches. A KG usually consists of a set of facts. A fact is a triple (head, relation, tail) where heads and tails are called entities. Among the SRL models, distance-based KG embeddings are popular because of their simplicity, their low number of parameters, and their efficiency on large scale datasets. Specifically, their simplicity allows integrating them into many models. Previous studies have integrated them with logical rule embeddings , have adopted them to encode temporal information and have applied them to find equivalent entities between multi-language datasets . Soon after the introduction of the first multi-relational distance-based method TransE it was acknowledged that it is inefficient in learning of symmetric relations, since the norm of the representation vector for all the symmetric relations in the KG becomes close to zero. This means the model cannot distinguish well between different symmetric relations in a KG. To extend this model many variations are studied afterwards, e.g., TransH (b), TransR (b), TransD , and STransE . Even though they solved the issue of symmetric relations, they introduced a new problem: these models were no longer efficient in learning the inversion and composition relation patterns that originally TransE could handle. Besides, as noted in , within the family of distancebased embeddings, usually reflexive relations are forced to become symmetric and transitive. In this study, we take advantage of independent vector representations of vectors that enable us to view the same relations from different aspects and put forward a translation-based model that addresses these limitations and allows the learning of all three relation patterns. In addition, we address the issue of the limit-based loss function in finding an optimal limit and suggest an updating limit loss function to be used complementary to the current limit-based loss function which has fixed limits. Moreover, we frame our model into a neural network structure that allows it to learn non-linear patterns between embedding vectors and the expected output which substantially improves the generalization power of the model in link prediction tasks. The model performs well in the empirical evaluations, improving upon the state-of-the-art in link prediction benchmarks. Since our approach involves several elements that model the relations between entities as the geometric distance of vectors from different views, we dubbed it multipledistance embeddings (MDE). Given the set of all entities E and the set of all relations R, we formally define a fact as a triple of the form (h, r, t) in which h is the head and t is the tail, h, t ∈ E and r ∈ R is a relation. A knowledge graph KG is a subset of all true facts KG ⊂ ζ and is represented by a set of triples. An embedding is a mapping from an entity or a relation to their latent representation. A latent representation is usually a (set of) vector(s), a matrix or a tensor of numbers. A relational learning model is made of an embedding function and a prediction function that given a triple (h, r, t) it determines if (h, r, t) ∈ ζ. We represent the embedding representation of an entity h with a lowercase letter h if it is a vector and with an uppercase letter H if it is a matrix. The ability to encode different patterns in the relations can show the generalization power of a model: Definition 1. A relation r is symmetric (antisymmetric) if ∀x, y r(x, y) ⇒ r(y, x) (r(x, y) ⇒ ¬r(y, x) ). A clause with such a structure has a symmetry (antisymmetry) pattern. Definition 2. A relation r 1 is inverse to relation r 2 if ∀x, y r 2 (x, y) ⇒ r 1 (y, x). A clause with such a form has an inversion pattern. Definition 3. A relation r 1 is composed of relation r 2 and relation r 3 if ∀x, y, z A clause with such a form has a composition pattern. Tensor Factorization and Multiplicative Models define the score of triples via pairwise multiplication of embeddings. DistMult simply multiplies the embedding vectors of a triple element by element h, r, t as the score function. Since multiplication of real numbers is symmetric, DistMult can not distinguish displacement of head relation and tail entities and therefore, it can not model anti-symmetric relations. ComplEx solves the issue of DistMult by the idea that the complex conjugate of the tail makes it non-symmetric. By introducing complex-valued embeddings instead of realvalued embeddings to DistMult, the score of a triple in ComplEx is Re(h diag(r)t) witht the conjugate of t and Re is the real part of a complex value. ComplEx is not efficient in encoding composition rules . In RESCAL instead of a vector, a matrix represents the relation r, and performs outer products of h and t vectors to this matrix so that its score function becomes h Rt. A simplified version of RESCAL is HolE that defines a vector for r and performs circular correlation of h and t has been found equivalent to ComplEx. Another tensor factorization model is Canonical Polyadic (CP) . In CP decomposition, each entity e is represented by two vectors h e, t e ∈ R d, and each relation r has a single embedding vector v r ∈ R d. MDE is similarly based on the idea of independent vector embeddings. A study suggests that in CP, the independence of vectors causes the poor performance of CP in KG completion, however, we show that the independent vectors can strengthen a model if they are combined complementarily. SimplE analogous to CP, trains on two sets of subject and object entity vectors. SimplE's score function, 1 2 h ei, r, t ej + 1 2 h ej, r −1, t ej, is the average of two terms. The first term is similar to DistMult. However, its combination with the second term and using a second set of entity vectors allows SimplE to avoid the symmetric issue of DistMult. SimplE allows learning of symmetry, anti-symmetry and inversion patterns. However, it is unable to efficiently encode composition rules, since it does not model a bijection mapping from h to t through relation r. In Latent Distance Approaches the score function is the distance between embedding vectors of entities and relations. In the view of social network analysis, originally proposed distance of entities −d(h, t) as the score function for modeling uni-relational graphs where d(., .) means any arbitrary distance, such as Euclidean distance. SE generalizes the distance for multi-relational data by incorporating a pair of relation matrices into it. TransE represents relation and entities of a triple by a vector that has this relation where. p is the p-norm. To better distinguish entities with complex relations, TransH (a) projects the vector of head and tail to a relation-specific hyperplane. Similarly, TransR follows the idea with relation-specific spaces and extends the distance function to M r h + r − M r t p. RotatE combines translation and rotation and defines the distance of a t from tail h which is rotated the amount r as the score function of a triple −d(h • r, t) where • is Hadamard product. Neural Network Methods train a neural network to learn the interaction of the h, r and t. ER-MLP is a two layer feedforward neural network considering h, r and t vectors in the input. NTN is neural tensor network that concatenates head h and tail t vectors and feeds them to the first layer that has r as weight. In another layer, it combines h and t with a tensor R that represents r and finally, for each relation, it defines an output layer r to represent relation embeddings. In SME relation r is once combined with the head h to get g u (h, r), and similarly it is combined with the tail t to get g v (t, r). SME defines a score function by the dot product of this two functions in the hidden layer. In the linear SME, g(e, r) is equal to M The score function of MDE involves multiple terms. We first explain the intuition behind each term and then explicate a framework that we suggest to efficiently utilize them such that we benefit from their strengths and avoid their weaknesses. Inverse Relation Learning: Inverse relations can be a strong indicator in knowledge graphs. For example, if IsP arentOf (m, c) represents that a person m is a parent of another person c, then this could imply IsChildOf (c, m) assuming that this represents the person c being the child of m. This indication is also valid in cases when this only holds in one direction, e.g. for the relations IsM otherOf and IsChildOf. In such a case, even though the actual inverse IsP arentOf may not even exist in the KG, we can still benefit from inverse relation learning. To learn the inverse of the relations, we define a score function S 2: Symmetric Relations Learning: It is possible to easily check that the formulation h + r − t allows 1 learning of anti-symmetric pattern but when learning symmetric relations, r tends toward zero which limits the ability of the model in separating entities specially if symmetric relations are frequent in the KG. For learning symmetric relations, we suggest the term S 3 as a score function. It learns such relations more efficiently despite it is limited in the learning of antisymmetric relations. Lemma 1. S 1 allows modeling antisymmetry, inversion and composition patterns and S 2 allows modeling symmetry patterns. (See proof in Appendix A) Relieving Limitations on Learning of Reflexive Relations: A previous study highlighted the common limitations of TransE, FTransE, STransE, TransH and TransR for learning reflexive relations where these translation-based models force the reflexive relations to become symmetric and transitive. To relieve these limitations, we define S 4 as a score function which is similar to the score of RotatE i.e., h • r − t p but with the Hadamard operation on the tail. In contrast to RotatE which represents entities as complex vectors, S 4 only holds in the real space: Lemma 2. The following restrictions of translation based embeddings approaches do not apply to the S 4 score function. R1: if a relation r is reflexive, on ∆ ∈ E, r it will be also symmetric on ∆. R2: if r is reflexive on ∆ ∈ E, r it will be also be transitive on ∆. (See proof in Appendix B) Model Definition: To incorporate different views to the relations between entities, we define these settings for the model: 1. Using limit-based loss instead of margin ranking loss. 2. Each aggregated term in the score represents a different view of entities and relations with an independent set of embedding vectors. 3. In contrast to ensemble approaches that incorporate models by training independently and testing them together, MDE is based on multi-objective optimization that jointly minimizes the objective functions. However, when aggregating different terms in the score function, the summation of opposite vectors can cause the norm of these vectors to diminish during the optimization. For example if S 1 and S 3 are added together, the minimization would lead to relation(r) vectors with zero norm value. To address this issue, we represent the same entities with independent variables in different distance functions. Based on CP, MDE considers four vectors e i, e j, e k, e l, ∈ R d as the embedding vector of each entity e, and four vectors r i, r j, r k, r l ∈ R d for each relation r. The score function of MDE for a triple (h, r, t) is defined as weighted sum of listed score functions: where ψ, w 1, w 2, w 3, w 4 ∈ R are constant values. In the following, we show using ψ and limitbased loss, the combination of the terms in equation 5 is efficient, such that if one of the terms recognises if a sample is true F M DE would also recognize it. Limit-based Loss: Because margin ranking loss minimizes the sum of error from directly comparing the score of negative to positive samples, when applying it to translation embeddings, it is possible that the score of a correct triplet is not small enough to hold the relation of the score function . To enforce the scores of positive triples become lower than those of negative ones, defines limited-based loss which minimizes the objective function such that Figure 1: Geometric illustration of the translation terms considered in MDE the score for all the positive samples become less than a fixed limit. extends the limit-based loss so that the score of the negative samples become greater than a fixed limit. We train our model with the same loss function which is: where − are the set of positive and negative samples and β 1, β 2 > 0 are constants denoting the importance of the positive and negative samples. This version of limit-based loss minimizes the aggregated error such that the score for the positive samples become less than γ 1 and the score for negative samples become greater than γ 2. To find the optimal limits for the limit-based loss, we suggest updating the limits during the training. (See the explanation in Appendix D). Lemma 3. There exist ψ and γ 1, γ 2 ≥ 0 (γ 1 ≥ γ 2), such that only if one of the terms in f M DE estimates a fact as true, f M DE also predicts it as a true fact. Consequently, the same also holds for the capability of MDE to allow learning of different relation patterns. (See proof in Appendix C) It is notable that without the introduction of ψ and the limits γ 1, γ 2 from the limit-based loss, Lemma 3 does not hold and framing the model with this settings makes the efficient combination of the terms in f M DE possible. In contrast to SimplE that ties the relation vectors of two terms in the score together, MDE does not directly relate them to take advantage of the independent relation and entity vectors in combining opposite terms. The learning of the symmetric relations is previously studied (e.g. in ) and (a) studied the training over the inverse of relations, however providing a way to gather all these benefits in one model is a novelty of MDE. Besides, complementary modeling of different vector-based views of a knowledge graph is a novel contribution. The score of MDE is already aggregating a multiplication of vectors to weights. We take advantage of this setting to model MDE as a layer of a neural network that allows learning the embedding vectors and multiplied weights jointly during the optimization. To create such a neural network we multiply ψ by a weight w 5 and we feed the MDE score to an activation function. We call this extension of MDE as MDE N N: where σ is logistic sigmoid function and w 1, w 2,..., w 5 are elements of the latent vector w that are estimated during the training of the model. This framing of MDE reduces the number of hyperparameters. The major advantage of MDE N N in comparison to the current distance-based models is that the logistic sigmoid activation function allows the non-linear mappings between the embedding vectors and the expected output for positive and the negative samples. Considering the ever growth of KGs and the expansion of the web, it is crucial that the time and memory complexity of a relational mode be minimal. Despite the limitations in expressivity, TransE is one of the popular models on large datasets due to its scalability. With O(d) time complexity (of one mini-batch), where d is the size of embedding vectors, it is more efficient than RESCAL, NTN, and the neural network models. Similar to TransE, the time complexity of MDE is O(d). Due to the additive construction of MDE, the inclusion of more distance terms keeps the time complexity linear in the size of vector embeddings. Datasets: We experimented on four standard datasets: WN18 and FB15k are extracted by from Wordnet Freebase . We used the same train/valid/test sets as in ( and the of TransR and NTN from , and ER-MLP from . The on the inverse relation excluded datasets are from , Table 13 for TransE and RotatE and the rest are from 2. Evaluation Settings: We evaluate the link prediction performance by ranking the score of each test triple against its versions with replaced head, and once for tail. Then we compute the hit at N (Hit@N), mean rank (MR) and mean reciprocal rank (MRR) of these rankings. We report the evaluations in the filtered setting. Implementation: We implemented MDE in PyTorch 3. Following , we generated one negative example per positive example for all the datasets. We used Adadelta as the optimizer and fine-tuned the hyperparameters on the validation dataset. The ranges of the hyperparameters are set as follows: embedding dimension 25, 50, 100, 200, batch size 100, 150, and iterations 50, 100, 1000, 1500, 2500, 3600. We set the initial learning rate on all datasets to 10. For MDE, the best embedding size and γ 1 and γ 2 and β 1 and β 2 values on WN18 were 50 and 1.9, 1.9, 2 and 1 respectively and for FB15k were 200, 10, 13, 1, 1. The best found embedding size and γ 1 and γ 2 and β 1 and β 2 values on FB15k-237 were 100, 9, 9, 1 and 1 respectively and for WN18RR were 50, 2, 2, 5 and 1. We selected the coefficient of terms in equation 5, by grid search in the range 0.1 to 1.0 and testing those combinations of the coefficients where they create a convex combination. Found values are w 1 = 0.16, w 2 = 0.33, w 3 = 0.16, w 4 =0.33. We also tested for the best value for ψ between {0.1, 0.2,. . ., 1.5}. We use ψ = 1.2 for all the experiments. For MDE N N, we use the same γ 1, γ 2, β 1 and β 2 values except for WN18 that the γ 1 and γ 2 are 4. We use the embedding size 50 for WN18RR, 200 for WN18, 200 for FB15k-237 and 200 for FB15k. We use ψ = 2 for all the MDE N N experiments. To regulate the loss function and to avoid over-fitting, we estimate the score function for two sets of independent vectors and we take their average in the prediction. Another advantage of this operation is the reduction of required training iterations. As a , MDE reaches to the 99 percent of its ranking performance in 100 iterations, and MDE N N reaches its best performance in the benchmarks in just 50 iterations. Table 2 shows the of our experiment on FB15k-237 and WN18RR, where the improvement is much more significant. Due to the existence of hard limits in the limit-based loss, the mean rank in both MDE and MDE N N is much lower than other methods. The comparison of MDE to other state-of-the-art models, regardless of the MDE N N, shows the competitive performance of MDE. It is observable that while MDE generates only one negative sample per positive sample and is using vector sizes between 50 to 200, it challenges RotatE which employs relatively large embedding dimensions (from 125 up to 1000) and high number of negative samples (up to 1024). We observe that the application of sigmoid in MDE N N improves it significantly in all the benchmarks. Particularly, in the more challenging tests over WN18RR and FB15k-237, the improvement is more significant. For example, we can see that the construction of the neural network from the model increased its Hit@10 on FB15k-237 from 0.484 to 0.999. From analyzing the MRR scores, we can see that RotatE must be totally off in few cases whereas the MDE N N model almost never seems to be far off, but frequently fails to put the correct entity on top. To our knowledge, MDE N N outperforms all the current embedding models in the MR and Hit@10 measures and specially performs better than all the existing models in all the measures on WN18RR and FB15k-237 benchmarks. To better understand the role of each term in the score function of MDE, we embark two ablation experiments. First, we train MDE using one of the terms alone, and observe the link prediction performance of each term in the filtered setting. In the second experiment, we remove one of the terms at a time and test the effect of the removal of that term on the model after 100 iterations. Table 3 summarizes the of the first experiment on WN18RR and FB15k-237. We can see that S 4 outperforms the other terms while S 1 and S 3 performs very similar on these two datasets. Between the four terms, S 2 performs the worst since most of the relations in the test datasets follow an antisymmetric pattern and S 2 is not efficient in modeling them. Table 4 shows the of the second experiment. The evaluations on WN18RR and WN18 show that removal of S 4 has the most negative effect on the performance of MDE. The removal of S 1 that was one of the good performing terms in the last experiment has the least effect. Nevertheless, S 1 improves the MRR in the MDE. Also, when we remove S 2, the MRR and Hit@10 are negatively influenced, indicating that there exist cases that S 2 performs better than the other terms, although, in the individual tests, it performed the worst between all the terms. In this study, we showed how MDE relieves the expressiveness restrictions of the distance-based embedding models and proposed a general method to override these limitations for the older models. Beside MDE and RotatE, most of the existing KG embedding approaches are unable to allow modeling of all the three relation patterns. We framed MDE into a Neural Network structure and validated our contributions via both theoretical proofs and empirical . We demonstrated that with multiple views to translation embeddings and using independent vectors (that previously were suggested to cause poor performance ) a model can outperform the existing state-of-the-art models for link prediction. Our experimental confirm the competitive performance of MDE and particularly MDE N N that achieves state-of-the-art MR and Hit@10 performance on all the benchmark datasets. A PROOF OF LEMMA 1. Let r 1, r 2, r 3 be relation vector representations and e i, e j, e k are entity representations. A relation r 1 between (e i, e k) exists when a triple (e i, r 1, e k) exists and we show it by r 1 (e i, e k). Formally, we have the following : Antisymmetric Pattern. If r 1 (e i, e j) and r 1 (e j, e i) hold, in equation 1 for S 1, then: e i + r 1 = e j ∧ e j + r 1 = e i ⇒ e i + 2r 1 = e i Therefore S 1 allows encoding of relations with antisymmetric patterns. Symmetric Pattern. If r 1 (e i, e j) and r 1 (e j, e i) hold, for S 2 we have: e i + e j − r 1 = 0 ∧ e j + e i − r 1 = 0 ⇒ e j + e i = r 1 Therefore S 2 allows encoding relations with symmetric patterns. For S 1 we have: Inversion Pattern. If r 1 (e i, e j) and r 2 (e j, e i) hold, from Equation 1 we have: Therefore S 1 allows encoding relations with inversion patterns. Composition Pattern. If r 1 (e i, e k), r 2 (e i, e j) and, r 3 (e j, e k) hold, from equation 1 we have: e i + r 1 = e k ∧ e i + r 2 = e j ∧ e j + r 3 = e k ⇒ r 2 + r 3 = r 1 Therefore S 1 allows encoding relations with composition patterns. B PROOF OF LEMMA 2. Proof. R1: For such reflexive r 1, if r 1 (e i, e i) then r l (e j, e j). In this equation we have: e i = r 1 e i ∧ e j = r 1 e j ⇒ r 1 = U ⇒ e i = r 1 e j where U is unit tensor. R2: For such reflexive r 1, if r 1 (e i, e j) and r l (e j, e k) then r 1 (e j, e i) and r l (e k, e j). In the above equation we have: e i = r 1 e j ∧ e j = r 1 e k ⇒ e i = r 1 r 1 e j e k ∧ r i = U ⇒ e i = e j e k ⇒ e i + e k = r l C PROOF OF LEMMA 3. We show there is boundries for γ 1, γ 2, w 1, w 2, w 3, w 4, such that learning a fact by one of the terms in f M DE is enough to classify a fact correctly. Proof. We show the boundaries for three aggregated terms in the the distance function, it is easily possible to extend it to four and more terms. It is enough to show that there is at least one set of boundaries for the positive and negative samples that follows the constraints. The case to prove is when three of the distance functions classify a fact negative N and the one distance function e.g. s 2 classify it as positive P, and the case that s 1 and s 3 classify a fact as positive and s 2 classify it as negative. We set w 1 = w 3 = 1/4 and w 2 = 1/2 and assume that Sum is the value estimated by the score function of MDE, we have: There exist a = 2 and γ 1 = γ 2 = 2 and ψ = 1 that satisfy γ 1 > Sum ≥ 0 and the inequality 8. loss = the from equation 6 It can be easily checked that without introduction of ψ, there is no value of Sum that can satisfy both γ 1 > Sum ≥ 0 and the inequality 8 and we calculated the value of ψ based on the values of γ 1, γ 2 and a. In case that future studies discover new interesting distances, this Lemma shows how to basically integrate them into MDE. While the limit-based loss resolves the issue of margin ranking loss with distance based embeddings, it does not provide a way to find the optimal limits. Therefore the mechanism to find limits for each dataset and hyper-parameter is the try and error. To address this issue, we suggest updating the limits in the limit-based loss function during the training iterations. We denote the moving-limit loss by loss guide. loss guide = lim δ,δ →γ1 where the initial value of δ 0, δ 0 is 0. In this formulation, we increase the δ 0, δ 0 toward γ 1 and γ 2 during the training iterations such that the error for positive samples minimizes as much as possible. We test on the validation set after each 50 epoch and take those limits that give the best value during the tests. The details of the search for limits is explained in Algorithm 1. After observing the most promising values for limits in the preset number of iterations, we stop the search and perform the training while having the δ values fixed(fixed limit-base loss) to allow the adaptive learning to reach loss values smaller than the threshold. We based this approach on the idea of adaptive learning rate , where the Adadelta optimizer adapts the learning rate after each iteration, therefore in the loss guided we can update the limits without stopping the training iterations. In our experiments, the variables in the Algorithm 1, are as follows. threshold = 0.05, ξ = 0.1.

Represent the short summary for finding the corresponding scientific article (long)

A novel method of modelling Knowledge Graphs based on Distance Embeddings and Neural Networks

Represent the TLDR for finding the corresponding scientific article!

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: Improved generative adversarial network (Improved GAN) is a successful method of using generative adversarial models to solve the problem of semi-supervised learning. However, it suffers from the problem of unstable training. In this paper, we found that the instability is mostly due to the vanishing gradients on the generator. To remedy this issue, we propose a new method to use collaborative training to improve the stability of semi-supervised GAN with the combination of Wasserstein GAN. The experiments have shown that our proposed method is more stable than the original Improved GAN and achieves comparable classification accuracy on different data sets. Generative adversarial networks (GANs) BID3 have been recently studied intensively and achieved great success in deep learning domain BID14 BID9 BID15. A typical GAN simulates a two-player minimax game, where one aims to fool the other and the overall system is finally able to achieve equilibrium. Specifically speaking, we have a generator G to generate fake data G(z) from a random variable z whose distribution density is p(z), and also we have a discriminator D(x) to discriminate the real x from the generated data G(z), where x ∼ p r (x) and p r is the distribution density of real data. We optimize the two players G(z) and D(x) by solving the following minimax problem: DISPLAYFORM0 This method is so called as the original GAN BID3. After this, many different types of GANs have been proposed, e.g., least-squared GAN BID9, cat-GAN BID15, W-GAN, Improved GAN BID14, so on and so forth, focusing on improving the performance of GANs and extending the GAN idea to other application scenarios. For instance, the original GAN is trained in a completely unsupervised learning way BID3, along with many variants, such as LS-GAN and cat-GAN. It was later extended to semi-supervised learning. In BID14, Salimans et al. proposed the Improved GAN to enable generation and classification of data simultaneously. In BID7, Li et al. extended this method to consider conditional data generation. Another issue regarding the unsupervised learning of GANs is the lack of training stability in the original GANs, mostly because of dimension mismatch. A lot of efforts have been dedicated to solve this issue. For instance, in, the authors theoretically found that the instability problem and dimension mismatch of the unsupervised learning GAN was due to the maxing out of Jensen-Shannon divergence between the true and fake distribution and therefore proposed using the Wasserstein distance to train GAN. However, to calculate the Wasserstein distance, the network functions are required to be 1-Lipschitz, which was simply implemented by clipping the weights of the networks in. Later, Gulrajani et. al. improved it by using gradient penalty BID4. Besides them, the same issue was also addressed from different perspectives. In BID13, Roth et al. used gradient norm-based regularization to smooth the f-divergence objective function so as to reduce dimension mismatch. However, the method could not directly work on f-divergence, which was intractable to solve, but they instead optimized its variational lower bound. Its converging rate is still an open question and its computational complexity may be high. On the other hand, there were also some efforts to solve the issue of mode collapse, so as to try to stabilize the training of GANs from another perspective, including the unrolled method in BID10, mode regularization with VAEGAN , and variance regularization with bi-modal Gaussian distributions BID5. However, all these methods were investigated in the context of unsupervised learning. Instability issue for semi-supervised GAN is still open. In this work, we focus on investigating the training stability issue for semi-supervised GAN. To the authors' best knowledge, it is the first work to investigate the training instability for semi-supervised GANs, though some were done for unsupervised GANs as aforementioned. The instability issue of the semi-supervised GAN BID14 is first identified and analyzed from a theoretical perspective. We prove that this issue is in fact caused by the vanishing gradients theorem on the generator. We thus propose to solve this issue by using collaborative training to improve its training stability. We theoretically show that the proposed method does not have vanishing gradients on the generator, such that its training stability is improved. Besides the theoretical contribution, we also show by experiments that the proposed method can indeed improve the training stability of the Improved GAN, and at the same time achieve comparable classification accuracy. It is also worth to note that BID7 proposed the Triple GAN that also possessed two discriminators. However, its purpose is focused on using conditional probability training (the original GAN uses unconditional probability) based on data labels to improve the training of GAN, but not on solving the instability issue. Therefore, the question of instability for the Triple GAN is still unclear. More importantly, the method, collaborative training, proposed for exploring the data labels with only unconditional probability in this paper, can also be applied to the Triple GAN to improve its training stability, in the case of conditional probability case. The rest of the paper is organized as follows: in Section 2, we present the generator vanishing gradient theorem of the Improved GAN. In Section 3, we propose a new method, collaborative training Wasserstein GAN (CTW-GAN) and prove its nonvanishing gradient theorem. In Section 4, we present our experimental and finally give our in Section 5. The improved GAN BID14 combines supervised and unsupervised learning to solve the semi-supervised classification problem by simulating a two-player minmax game with adversarial training. The adversarial training is split into two steps. In the first step, it minimizes the following objective function for discriminator D for data x and labels y: DISPLAYFORM0 In the second step, it minimizes the distance of feature matching to optimize the generator G: DISPLAYFORM1 where DISPLAYFORM2 and D (−3) (x) are the outputs from the (n − 3)-th layer for a net with n layers. In this subsection, we prove the theorem of vanishing gradients on the generator for Improved GAN. This explains why the Improved GAN lacks training stability, as showed on some datasets, such as MNIST (cf. Section 4).Theorem 2.1 (Vanishing gradients on the generator for Improved GAN) Let g θ: Z → X be a differentiable function that induces a distribution P g. Let P r be the real data distribution. Let D be a differentiable discriminator bounded by T, i.e., D(x) 2 ≤ T. If the discriminator is trained to converge, i.e., D − D * 2 <, and DISPLAYFORM0 Proof 2.1 See Appendix A.1.This theorem implies that the generator gradients vanish when the discriminator is trained to converge. In this case, the generator training saturates, which explains the training instability phenomenon of the Improved GAN. The way to solve this problem is our next question. In this section, we propose a new method to solve the instability issue of the Improved GAN by using collaborative training between two GANs. These two GANs contribute to the adversarial training from two different perspectives, which may help avoid the drawbacks of each one. This is the basic idea behind the proposed method. The detailed procedure of CTW-GAN can be summarized as a minimax game carried out in two steps:At the first step, the discriminators D c and D w are optimized simultaneously: DISPLAYFORM0 where DISPLAYFORM1 At the second step, the generator G is then optimized by applying the optimized two discriminators D c, D w to G: DISPLAYFORM2 where DISPLAYFORM3 The overall architecture for CTW-GAN is described by Figure 1, where x r and x u stand for labeled and unlabeled data respectively:Figure 1: Architecture for CTW-GAN Bearing in mind the generator vanishing gradients theorem for Improved GAN, we may ask if a similar problem exists for our proposed CTW-GAN. In the following, we prove that our proposed method does not have the vanishing gradients issue on the generator, which therefore improves the training stability of the original Improved GAN.Theorem 3.1 Let P r be any distribution. Let P θ be the distribution of g θ (z) with z being a random variable with a density p and g θ a continuous function with respect to θ. Then there is a set of solutions D c, D w to the problem DISPLAYFORM0 and we have DISPLAYFORM1 where the last term is the gradients of the Wasserstein distance W (P r, P θ), i.e., DISPLAYFORM2 when the term L(D c, D w) is well-defined. Proof 3.1 see Appendix A.2.Remark: the above D w L ≤ 1 is required to be 1-Lipschitz. The constraint can be realized by weight clipping or gradient penalty BID4.The proposed algorithm is described as follows:Algorithm 1 CTW-GAN with gradient penalty: Require: Gradient penalty λ p = 10, generator weightλ g, and Adam hyperparameter α = 0.0001 Require: Initial parameter θ w for D w, θ c for D c and θ g for G.while θ g does not converge do for i = 1 · · · m do Sample a real data x ∼ p r and a noisy data DISPLAYFORM3 Sample a real data x ∼ p r, a noisy data z ∼ p(z) and a random variable ∼ U DISPLAYFORM4 In this section, we shall present the experiments to evaluate the proposed method. Our evaluation goals are twofold. On one hand, we evaluate the stability of CTW-GAN in comparison to that of the original Improved GAN to see whether our proposed method improves the training stability or not. On the other hand, we evaluate whether the proposed method achieves comparable classification performance to the original Improved GAN. To this end, we run experiments on two datasets: MNIST and CIFAR-10. MNIST includes 50, 000 training images, 10, 000 validation images and 10, 000 testing images, which contain handwritten digits with size 28 × 28.Following BID14, we randomly select a small set of labeled data from the 60, 000 training and validation set to perform semi-supervised learning with the selection size of 20, 50, 100, and 200 labeled examples. We run our experiments 9 times by giving the program different seeds. We use the seeds from 1 − 9. For each seed, the labeled data is selected so as to have a balanced number of examples from each class. The rest of the training images are used as unlabeled data. In our method, we use three networks whose architectures are described in FIG0. We use batch normalization and add Gaussian noise to the output of each layer of the two discriminators as the original Improved GAN does BID14. We only tune the parameter λ = 0.1, 0.5 from two values on the MNIST dataset. We do not tune any other parameters, such as learning rate, step size, etc.: we keep these as in the original Improved GAN. The shown in TAB1 are reported with λ = 0.1, the threshold for gradient penalty is 10 and n critic = 5: DISPLAYFORM0 From the , we can easily see that the original improved GAN has one or two out of nine runs for training failure (unexpected high error rates and poor generate image quality). However, for our proposed method, no training failure occurs. This shows that our method improves the training stability indeed. On the other hand, besides making the training process more stable, our proposed method does not reduce the classification accuracy at all, which is beyond our original purpose of avoiding training instability of the Improved GAN. Reasoning it, it may imply that the information explored by the two discriminators may be very different, thus reflecting a distinct Method n=50 n=100 n=200 DGN BID6 3.33(±0.14) Virtual Adversarial BID11 2.12 Cat-GAN BID15 1.91 (±0.10) Skip Deep Generative Model BID8 1.32 (±0.07) Ladder network BID12 1.06 (±0.37) Auxiliary Deep Generative Model BID8 0 GAN 20.40 (±0.47) Ladder network BID12 19.58 (±0.46) Improved-GAN 0.1726 (±0.0032) Ours 0.1713 (±0.0014) There is no failure case found in three runs for the original GAN on CIFAR-10. We use 4000 labeled samples.source of information for data representation. Utilizing those different information sources may help to improve classification accuracy, as long as the source of information is meaningful to some extent, or at least not noise. In our method, we use a very simple network for D w with only two layers. It may be possible to further improve classification performance if a network with more layers is used. We leave it for future work. In this section, we test our proposed method on the data set of CIFAR-10. CIFAR-10 consists of colored images belonging to 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship and truck. There are 50,000 training and 10,000 testing samples with the size of 32 × 32. We split 5,000 training data of CIFAR-10 for validation if needed. Following BID14, we use a 9 layer deep convolutional network with dropout and weight normalization for the discriminator D c. The generator G is a 4 layer deep CNN with batch normalization. We use a very simple network with three layers for the discriminator D w, due to the limiting GPU resource. The network architectures are given in FIG1. TAB2 summarizes our on the semi-supervised learning task. On CIFAR-10 dataset, it is interesting to see that there is no failure case found for the Improved GAN in three runs at the moment. From the theoretical viewpoint, this may be due to the abundant richness of the image features in color being much harder to be modeled by the neural nets than that of MNIST in grayscale. Thus, the discriminator D c trained on CIFAR-10 does not as easily converge as the one trained on MNIST, such that the gradients on the generator do not vanish. However, it does not mean that this possibility is avoided. In certain cases, as long as the discriminator is trained to converge., e.g., running more iterations than the generator, the gradients on the generator will surely vanish, as theoretically guaranteed by Theorem 2.1. On the other hand, our proposed method is still able to achieve comparable to the original Improved GAN, besides providing a theoretical guarantee to the training stability. Due to the limiting GPU resource, we use a very simple network for D w. In this sense, the characteristics captured by this network may not be rich enough. However, the showed that even with the very simple network, the classification performance obtained is roughly comparable to that of the Improved GAN. We expect that it would be possibly improved further if we have more GPU resources and are able to train a deeper network for D w. In the paper, we study the training instability issue of semi-supervised improved GAN. We have found that the training instability is mainly due to the vanishing gradients on the generator of the Improved GAN. In order to make the training of the Improved GAN more stable, we propose a collaborative training method to combine Wasserstein GAN with the semi-supervised improved GAN. Both theoretical analysis and experimental on MNIST and CIFAR-10 have shown the effectiveness of the proposed method to improve training stability of the Improved GAN. In addition, it also achieves the classification accuracy comparable to the original Improved GAN. We would like to thank National Natural Science Foundation of China FORMULA0 for previously supporting the authors to prepare for the knowledge and skills demanded by this work.

Improve Training Stability of Semi-supervised Generative Adversarial Networks with Collaborative Training

Represent the TLDR for finding the corresponding scientific article

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.