Question: This paper proposes a method for efficient training of Q-function for continuous-state Markov Decision Processes (MDP), such that the traces of the ing policies satisfy a Linear Temporal Logic (LTL) property. LTL, a modal logic, can express a wide range of time-dependent logical properties including safety and liveness. We convert the LTL property into a limit deterministic Buchi automaton with which a synchronized product MDP is constructed. The control policy is then synthesised by a reinforcement learning algorithm assuming that no prior knowledge is available from the MDP. The proposed method is evaluated in a numerical study to test the quality of the generated control policy and is compared against conventional methods for policy synthesis such as MDP abstraction (Voronoi quantizer) and approximate dynamic programming (fitted value iteration). Markov Decision Processes (MDPs) are extensively used as a family of stochastic processes in automatic control, computer science, economics, etc. to model sequential decision-making problems. Reinforcement Learning (RL) is a machine learning algorithm that is widely used to train an agent to interact with an MDP when the stochastic behaviour of the MDP is initially unknown. However, conventional RL is mostly focused on problems in which MDP states and actions are finite. Nonetheless, many interesting real-world tasks, require actions to be taken in response to high-dimensional or real-valued sensory inputs BID5. For example, consider the problem of drone control in which the drone state is represented as its Euclidean position (x, y, z) ∈ R 3.Apart from state space discretisation and then running vanilla RL on the abstracted MDP, an alternative solution is to use an approximation function which is achieved via regression over the set of samples. At a given state, this function is able to estimate the value of the expected reward. Therefore, in continuous-state RL, this approximation replaces conventional RL state-action-reward look-up table which is used in finite-state MDPs. A number of methods are available to approximate the expected reward, e.g. CMACs BID34, kernel-based modelling BID22, tree-based regression BID7, basis functions BID3, etc. Among these methods, neural networks offer great promise in reward modelling due to their ability to approximate any non-linear function BID13. There exist numerous successful applications of neural networks in RL for infinite or large-state space MDPs, e.g. Deep Q-networks BID19, TD-Gammon BID36, Asynchronous Deep RL BID20, Neural Fitted Q-iteration BID26, CACLA BID39.In this paper, we propose to employ feedforward networks (multi-layer perceptrons) to synthesise a control policy for infinite-state MDPs such that the generated traces satisfy a Linear Temporal Logic (LTL) property. LTL allows to specify complex mission tasks in a rich time-dependent formal language. By employing LTL we are able to express complex high-level control objectives that are hard to express and achieve for other methods from vanilla RL BID35 BID31 to more recent developments such as Policy Sketching BID1. Examples include liveness and cyclic properties, where the agent is required to make progress while concurrently executing components, to take turns in critical sections or to execute a sequence of tasks periodically. The purpose of this work is to show that the proposed architecture efficiently performs and is compatible with RL algorithms that are core of recent developments in the community. Unfortunately, in the domain of continuous-state MDPs, to the best of our knowledge, no research has been done to enable RL to generate policies according to full LTL properties. On the other hand, the problem of control synthesis in finite-state MDPs for temporal logic has been considered in a number of works. In BID41, the property of interest is an LTL property, which is converted to a Deterministic Rabin Automaton (DRA). A modified Dynamic Programming (DP) algorithm is then proposed to maximise the worst-case probability of satisfying the specification over all transition probabilities. Notice that in this work the MDP must be known a priori. BID8 and BID2 assume that the given MDP has unknown transition probabilities and build a Probably Approximately Correct MDP (PAC MDP), which is producted with the logical property after conversion to DRA. The goal is to calculate the value function for each state such that the value is within an error bound of the actual state value where the value is the probability of satisfying the given LTL property. The PAC MDP is generated via an RL-like algorithm and standard value iteration is applied to calculate the values of states. Moving away from full LTL logic, scLTL is proposed for mission specification, with which a linear programming solver is used to find optimal policies. The concept of shielding is employed in BID0 to synthesise a reactive system that ensures that the agent stays safe during and after learning. However, unlike our focus on full LTL expressivity, BID0 adopted the safety fragment of LTL as the specification language. This approach is closely related to teacher-guided RL BID37, since a shield can be considered as a teacher, which provides safe actions only if absolutely necessary. The generated policy always needs the shield to be online, as the shield maps every unsafe action to a safe action. Almost all other approaches in safe RL either rely on ergodicity of the underlying MDP, e.g. , which guarantees that any state is reachable from any other state, or they rely on initial or partial knowledge about the MDP, e.g. BID32 and BID17 ). Definition 2.1 (Continuous-state Space MDP) The tuple M = (S, A, s 0, P, AP, L) is an MDP over a set of states S = R n, where A is a finite set of actions, s 0 is the initial state and P: DISPLAYFORM0 ] is a Borel-measurable transition kernel which assigns to any state and any action a probability measure on the Borel space (R n, B(R n)) BID6. AP is a finite set of atomic propositions and a labelling function L: S → 2 AP assigns to each state s ∈ S a set of atomic propositions L(s) ⊆ 2 AP BID6.A finite-state MDP is a special case of continuous-state space MDP in which |S| < ∞ and P: DISPLAYFORM1 is the transition probability function. The transition function P induces a matrix which is usually known as transition probability matrix in the literature. Theorem 2.1 In any MDP M with bounded reward function and finite action space, if there exists an optimal policy, then that policy is stationary and deterministic. BID24 BID4.An MDP M is said to be solved if the agent discovers an optimal policy Pol *: S → A to maximize the expected reward. From Definitions A.3 and A.4 in Appendix, it means that the agent has to take actions that return the highest expected reward. Note that the reward function for us as the designer is known in the sense that we know over which state (or under what circumstances) the agent will receive a given reward. The reward function specifies what the agent needs to achieve but not how to achieve it. Thus, the objective is that the agent itself comes up with an optimal policy. In the supplementary materials in Section A.2, we present fundamentals of approaches introduced in this paper for solving infinite-state MDPs. In order to specify a set of desirable constraints (i.e. properties) over the agent policy we employ Linear Temporal Logic (LTL) BID23 ). An LTL formula can express a wide range of properties, such as safety and persistence. LTL formulas over a given set of atomic propositions AP are syntactically defined as DISPLAYFORM0 We define the semantics of LTL formula next, as interpreted over MDPs. Given a path ρ, the i-th state of ρ is denoted by DISPLAYFORM1 Definition 3.1 (LTL Semantics) For an LTL formula ϕ and for a path ρ, the satisfaction relation ρ |= ϕ is defined as DISPLAYFORM2.] |= ϕ 1 Using the until operator we are able to define two temporal modalities: eventually, ♦ϕ = true∪ϕ; and always, ϕ = ¬♦¬ϕ. LTL extends propositional logic with the temporal modalities until ∪, eventually ♦, and always. For example, in a robot control problem, statements such as "eventually get to this point" or "always stay safe" are expressible by these modalities and can be combined via logical connectives and nesting to provide general and complex task specifications. Any LTL task specification ϕ over AP expresses the following set of words: DISPLAYFORM3 Definition 3.2 (Policy Satisfaction) We say that a stationary deterministic policy Pol satisfies an LTL formula ϕ if: DISPLAYFORM4 For an LTL formula ϕ, an alternative method to express the set of associated words, i.e., W ords(ϕ), is to employ an automaton. Limit Deterministic Büchi Automatons (LDBA) are one of the most succinct and simplest automatons for that purpose. We need to first define a Generalized Büchi Automaton (GBA) and then we formally introduce an LDBA. Definition 3.3 (Generalized Büchi Automaton) A GBA N = (Q, q 0, Σ, F, ∆) is a structure where Q is a finite set of states, q 0 ⊆ Q is the set of initial states, Σ = 2 AP is a finite alphabet, F = {F 1, ..., F f} is the set of accepting conditions where F j ⊂ Q, 1 ≤ j ≤ f, and ∆: Q × Σ → 2 Q is a transition relation. Let Σ ω be the set of all infinite words over Σ. An infinite word w ∈ Σ ω is accepted by a GBA N if there exists an infinite run θ ∈ Q ω starting from q 0 where DISPLAYFORM5 where inf (θ) is the set of states that are visited infinitely often in the sequence θ. ): DISPLAYFORM6 • ∆(q, α) ⊆ Q D and |∆(q, α)| = 1 for every state q ∈ Q D and for every corresponding α ∈ Σ, DISPLAYFORM7 An LDBA is a GBA that has two partitions: initial (Q N) and accepting (Q D). The accepting part includes all the accepting states and has deterministic transitions. In this section, we propose an algorithm based on Neural Fitted Q-iteration (NFQ) that is able to synthesize a policy that satisfies a temporal logic property. We call this algorithm LogicallyConstrained NFQ (LCNFQ). We relate the notion of MDP and automaton by synchronizing them to create a new structure that is first of all compatible with RL and second that embraces the logical property. DISPLAYFORM0 ⊗ is the set of accepting states such that for each s ⊗ = (s, q) ∈ F ⊗, q ∈ F. The intuition behind the transition kernel P ⊗ is that given the current state (s i, q i) and action a the new state is (s j, q j) where s j ∼ P (·|s i, a) and q j ∈ ∆(q i, L(s j)).By constructing the product MDP we add an extra dimension to the state space of the original MDP. The role of the added dimension is to track the automaton state and, hence, to synchronize the current state of the MDP with the state of the automaton and thus to evaluate the satisfaction of the associated LTL property. Definition 4.2 (Absorbing Set) We define the set A ∈ B(S ⊗) to be an absorbing set if P ⊗ (A|s ⊗, a) = 1 for all s ⊗ ∈ A and for all a ∈ A. An absorbing set is called accepting if it includes F ⊗. We denote the set of all accepting absorbing sets by A.Note that the defined notion of absorbing set in continuous-state MDPs is equivalent to the notion of maximum end components in finite-state MDPs. In another word, once a trace ends up in an absorbing set (or a maximum end component) it can never escape from it BID38.The product MDP encompasses transition relations of the original MDP and the structure of the Büchi automaton, thus it inherits characteristics of both. Therefore, a proper reward function can lead the RL agent to find a policy that is optimal and that respects both the original MDP and the LTL property ϕ. In this paper, we propose an on-the-fly random variable reward function that observes the current state s ⊗, the action a and observes the subsequent state s ⊗ and gives the agent a scalar value according to the following rule: DISPLAYFORM1 where DISPLAYFORM2 is a positive reward and r n = y × m × rand (s ⊗) is a neutral reward where y ∈ {0, 1} is a constant, 0 < m M, and rand: S ⊗ → is a function that generates a random number in for each state s ⊗ each time R is being evaluated. The role of function rand is to break the symmetry in LCNFQ neural nets. Note that parameter y essentially acts as a switch to bypass the effect of the rand function on R. As we will see later, this switch is only active for LCNFQ.In LCNFQ, the temporal logic property is initially specified as a high-level LTL formula ϕ. The LTL formula is then converted to an LDBA N to form a product MDP M N (see Definition 4.1). In order to use the experience replay technique we let the agent explore the MDP and reinitialize it when a positive reward is received or when no positive reward is received after th iterations. The parameter th is set manually according to the MDP such that allows the agent to explore the MDP and also to prevent the sample set to explode in size. All episode traces, i.e. experiences, are stored in the form of (s DISPLAYFORM3 is the current state in the product MDP, a is the chosen action, s ⊗ = (s, q) is the ing state, and R(s ⊗, a) is the reward. The set of past experiences is called the sample set E.Once the exploration phase is finished and the sample set is created, we move forward to the learning phase. In the learning phase, we employ n separate multi-layer perceptrons with just one hidden layer where n = |Q| and Q is the finite cardinality of the automaton N. Each neural net is associated with a state in the LDBA and together the neural nets approximate the Q-function in the product MDP. For each automaton state q i ∈ Q the associated neural net is called B qi: S ⊗ × A → R. Once the agent is at state s ⊗ = (s, q i) the neural net B qi is used for the local Q-function approximation. The set of neural nets acts as a global hybrid Q-function approximator Q: S ⊗ × A → R. Note that the neural nets are not fully decoupled. For example, assume that by taking action a in state s ⊗ = (s, q i) the agent is moved to state s ⊗ = (s, q j) where q i = q j. According to the weights of B qi are updated such that B qi (s ⊗, a) has minimum possible error to R(s DISPLAYFORM4 Therefore, the value of DISPLAYFORM5 Let q i ∈ Q be a state in the LDBA. Then define E qi := {(·, ·, ·, ·, x) ∈ E|x = q i } as the set of experiences within E that is associated with state q i, i.e., E qi is the projection of E onto q i. Once the Algorithm 1: LCNFQ input:MDP M, a set of transition samples E output:Approximated Q-function 1 initialize all neural nets Bq i with (s0, qi, a) as the input and rn as the output where a ∈ A is a random action 2 repeat 3 for qi = |Q| to 1 do DISPLAYFORM6 end 10 until end of trial experience set E is gathered, each neural net B qi is trained by its associated experience set E qi. At each iteration a pattern set P qi is generated based on E qi: DISPLAYFORM7 The pattern set is used to train the neural net B qi. We use Rprop BID27 to update the weights in each neural net, as it is known to be a fast and efficient method for batch learning BID26. In each cycle of LCNFQ (Algorithm 1), the training schedule starts from networks that are associated with accepting states of the automaton and goes backward until it reaches the networks that are associated to the initial states. In this way we allow the Q-value to back-propagate through the networks. LCNFQ stops when the generated policy satisfies the LTL property and stops improving for long enough. Remark 4.1 We tried different embeddings such as one hot encoding BID11 and integer encoding in order to approximate the global Q-function with a single feedforward net. However, we observed poor performance since these encoding allows the network to assume an ordinal relationship between automaton states. Therefore, we turned to the final solution of employing n separate neural nets that work together in a hybrid manner to approximate the global Q-function. Recall that the reward function only returns a positive value when the agent has a transition to an accepting state in the product MDP. Therefore, if accepting states are reachable, by following this reward function the agent is able to come up with a policy Pol ⊗ * that leads to the accepting states. This means that the trace of read labels over S (see Definition 4.1) in an automaton state to be accepting. Therefore, the trace over the original MDP is a trace that satisfies the given logical property. Recall that the optimal policy has the highest expected reward comparing to other policies. Consequently, the optimal policy has the highest expected probability of reaching to the accepting set, i.e. satisfying the LTL property. The next section studies state space discretization as the most popular alternative approach to solving infinite-state MDPs. Inspired by BID15, we propose a version of Voronoi quantizer that is able to discretize the state space of the product MDP S ⊗. In the beginning, C is initialized to consist of just one c 1, which corresponds to the initial state. This means that the agent views the entire state space as a homogeneous region when no apriori knowledge is available. Subsequently, when the agent explores, the Euclidean distance between each newly visited state and its nearest neighbor is calculated. If this distance is greater than a threshold value ∆ called "minimum resolution", or if the new state s ⊗ has a never-visited automaton state then the newly visited state is appended to C. Therefore, as the agent continues to explore, the size of C would increase until the relevant parts of the state space are DISPLAYFORM0 || 2 } is defined by the nearest neighbor rule for any i = i. The VQ algorithm is presented in Algorithm 2. The proposed algorithm consist of several resets at which the agent is forced to re-localize to its initial state s 0. Each reset is called an episode, as such in the rest of the paper we call this algorithm episodic VQ. In this section we propose a modified version of FVI that can handle the product MDP. The global value function v: S ⊗ → R, or more specifically v: S × Q → R, consists of n number of sub-value functions where n = |Q|. For each q j ∈ Q, the sub-value function v qj: S → R returns the value the states of the form (s, q j). As we will see shortly, in a same manner as LCNFQ, the sub-value functions are not decoupled. Let P ⊗ (dy|s ⊗, a) be the distribution over S ⊗ for the successive state given that the current state is s ⊗ and the current action is a. For each state (s, q j), the Bellman update over each sub-value function v qj is defined as: DISPLAYFORM0 where T is the Bellman operator BID12. The update in is a special case of general Bellman update as it does not have a running reward and the (terminal) reward is embedded via value function initialization. The value function is initialized according to the following rule: DISPLAYFORM1 Algorithm 3: FVI input:MDP M, a set of samples {s DISPLAYFORM2 for each qj ∈ Q, Monte Carlo sampling number Z, smoothing parameter h output :approximated value function Lv 1 initialize Lv 2 sample Y Z a (si, qj), ∀qj ∈ Q, ∀i = 1,..., k, ∀a ∈ A 3 repeat 4 for j = |Q| to 1 do 5 ∀qj ∈ Q, ∀i = 1,..., k, ∀a ∈ A calculate Ia((si, qj)) = 1/Z y∈Y Z a (s i,q j) Lv(y) using FORMULA23 6 for each state (si, qj), update v q j (si) = sup a∈A {Ia((si, qj))} in 7 end 8 until end of trial where r p and r n are defined in. The main hurdle in executing the Bellman operator in continuous state MDPs, as in FORMULA19, is that no analytical representation of the value function v and also sub-value functions v qj, q j ∈ Q is available. Therefore, we employ an approximation method by introducing the operator L. The operator L constructs an approximation of the value function denoted by Lv and of each sub-value function v qj which we denote by Lv qj. For each q j ∈ Q the approximation is based on a set of points {( DISPLAYFORM3 ⊗ which are called centers. For each q j, the centers i = 1, ..., k are distributed uniformly over S such that they uniformly cover S.We employ a kernel-based approximator for our FVI algorithm. Kernel-based approximators have attracted a lot of attention mostly because they perform very well in high-dimensional state spaces BID33 . One of these methods is the kernel averager in which for any state (s, q j) the approximate value function is represented by DISPLAYFORM4 where the kernel K: S → R is a radial basis function, such as e −|s−si|/h, and h is smoothing parameter. Each kernel has a center s i and the value of it decays to zero as s diverges from s i. This means that for each q j ∈ Q the approximation operator L is a convex combination of the values of the centers {s i} k i=1 with larger weight given to those values v qj (s i) for which s i is close to s. Note that the smoothing parameter h controls the weight assigned to more distant values (see Section A.3).In order to approximate the integral in Bellman update we use a Monte Carlo sampling technique BID28 ). For each center (s i, q j) and for each action a, we sample the next state y z a (s i, q j) for z = 1,..., Z times and append it to set of Z subsequent states Y Z a (s i, q j). We then replace the integral with DISPLAYFORM5 The approximate value function Lv is initialized according to. In each cycle of FVI, the approximate Bellman update is first performed over the sub-value functions that are associated with accepting states of the automaton, i.e. those that have initial value of r p, and then goes backward until it reaches the sub-value functions that are associated to the initial states. In this manner, we allow the state values to back-propagate through the transitions that connects the sub-value function via. Once we have the approximated value function, we can generate the optimal policy by following the maximum value (Algorithm 3). We describe a mission planning architecture for an autonomous Mars-rover that uses LCNFQ to follow a mission on Mars. The scenario of interest is that we start with an image from the surface of Mars and then we add the desired labels from 2 AP, e.g. safe or unsafe, to the image. We assume that we know the highest possible disturbance caused by different factors (such as sand storms) on the rover motion. This assumption can be set to be very conservative given the fact that there might be some unforeseen factors that we did not take into account. The next step is to express the desired mission in LTL format and run LCNFQ on the labeled image before sending the rover to Mars. We would like the rover to satisfy the given LTL property with the highest probability possible starting from any random initial state (as we can not predict the landing location exactly). Once LCNFQ is trained we use the network to guide the rover on the Mars surface. We compare LCNFQ with Voronoi quantizer and FVI and we show that LCNFQ outperforms these methods. In this numerical experiment the area of interest on Mars is Coprates quadrangle, which is named after the Coprates River in ancient Persia (see Section A.4). There exist a significant number of signs of water, with ancient river valleys and networks of stream channels showing up as sinuous and meandering ridges and lakes. We consider two parts of Valles Marineris, a canyon system in Coprates quadrangle FIG2. The blue dots, provided by NASA, indicate locations of recurring slope lineae (RSL) in the canyon network. RSL are seasonal dark streaks regarded as the strongest evidence for the possibility of liquid water on the surface of Mars. RSL extend downslope during a warm season and then disappear in the colder part of the Martian year BID18. The two areas mapped in FIG2, Melas Chasma and Coprates Chasma, have the highest density of known RSL.For each case, let the entire area be our MDP state space S, where the rover location is a single state s ∈ S. At each state s ∈ S, the rover has a set of actions A = {left, right, up, down, stay} by which it is able to move to other states: at each state s ∈ S, when the rover takes an action a ∈ {left, right, up, down} it is moved to another state (e.g., s) towards the direction of the action with a range of movement that is randomly drawn from (0, D] unless the rover hits the boundary of the area which forces the rover to remain on the boundary. In the case when the rover chooses action a = stay it is again moved to a random place within a circle centered at its current state and with radius d D. Again, d captures disturbances on the surface of Mars and can be tuned accordingly. With S and A defined we are only left with the labelling function L: S → 2 AP which assigns to each state s ∈ S a set of atomic propositions L(s) ⊆ 2AP. With the labelling function, we are able to divide the area into different regions and define a logical property over the traces that the agent generates. In this particular experiment, we divide areas into three main regions: neutral, unsafe and target. The target label goes on RSL (blue dots), the unsafe label lays on the parts with very high elevation (red coloured) and the rest is neutral. In this example we assume that the labels do not overlap each other. Note that when the rover is deployed to its real mission, the precise landing location is not known. Therefore, we should take into account the randomness of the initial state s 0. The dimensions of the area of interest in FIG2.a are 456.98 × 322.58 km and in FIG2 DISPLAYFORM0 The first control objective in this numerical example is expressed by the following LTL formula over Melas Chasma FIG2: DISPLAYFORM0 where n stands for "neutral", t 1 stands for "target 1", t 2 stands for "target 2" and u stands for "unsafe". Target 1 are the RSL (blue bots) on the right with a lower risk of the rover going to unsafe region and the target 2 label goes on the left RSL that are a bit riskier to explore. Conforming to the rover has to visit the target 1 (any of the right dots) at least once and then proceed to the target 2 (left dots) while avoiding unsafe areas. Note that according to (u → u) in the agent is able to go to unsafe area u (by climbing up the slope) but it is not able to come back due to the risk of falling. With FORMULA26 we can build the associated Büchi automaton as in FIG3.a. The second formula focuses more on safety and we are going to employ it in exploring Coprates Chasma (FIG2 where a critical unsafe slope exists in the middle of this region. DISPLAYFORM1 In, t refers to "target", i.e. RSL in the map, and u stands for "unsafe". According to this LTL formula, the agent has to eventually reach the target (♦t) and stays there ((t → t)). However, if the agent hits the unsafe area it can never comes back and remains there forever ((u → u)). With we can build the associated Büchi automaton as in FIG3.b. Having the Büchi automaton for each formula, we are able to use Definition 4.1 to build product MDPs and run LCNFQ on both. This section presents the simulation . All simulations are carried on a machine with a 3.2GHz Core i5 processor and 8GB of RAM, running Windows 7. LCNFQ has four feedforward neural networks for and three feedforward neural networks for, each associated with an automaton state in FIG3.a and FIG3.b. We assume that the rover lands on a random safe place and has to find its way to satisfy the given property in the face of uncertainty. The learning discount factor γ is also set to be equal to 0.9. Fig. 4 in Section A.5 gives the of learning for LTL formulas FORMULA26 and FORMULA27. At each state s ⊗, the robot picks an action that yields highest Q(s ⊗, ·) and by doing so the robot is able to generate a control policy Pol ⊗ * over the state space S ⊗. The control policy Pol ⊗ * induces a policy Pol * over the state space S and its performance is shown in Fig. 4.Next, we investigate the episodic VQ algorithm as an alternative solution to LCNFQ. Three different resolutions (∆ = 0.4, 1.2, 2 km) are used to see the effect of the resolution on the quality of the generated policy. The are presented in TAB1, where VQ with ∆ = 2 km fails to find a satisfying policy in both regions, due to the coarseness of the ed discretisation. A coarse partitioning in the RL not to be able to efficiently back-propagate the reward or the agent to be stuck in some random-action loop as sometimes the agent's current cell is large enough that all actions have the same value. In TAB1, training time is the empirical time that is taken to train the algorithm and travel distance is the distance that agent traverses from initial state to final state. We show the generated policy for ∆ = 1.2 km in Fig. 5 in Section A.5. Additionally, Fig. 7 in Section A.6 depicts the ed Voronoi discretisation after implementing the VQ algorithm. Note that with VQ only those parts of the state space that are relevant to satisfying the property are accurately partitioned. Finally, we present the of FVI method in Fig 6 in Section A.5 for the LTL formulas and. The FVI smoothing parameter is h = 0.18 and the sampling time is Z = 25 for both regions where both are empirically adjusted to have the minimum possible value for FVI to generate satisfying policies. The number of basis points also is set to be 100, so the sample complexity of FVI is 100 × Z × |A| × (|Q| − 1). We do not sample the states in the product automaton that are associated to the accepting state of the automaton since when we reach the accepting state the property is satisfied and there is no need for further exploration. Hence, the last term is (|Q| − 1). However, if the property of interest produces an automaton that has multiple accepting states, then we need to sample those states as well. Note that in TAB1, in terms of timing, FVI outperforms the other methods. However, we have to remember that FVI is an approximate DP algorithm, which inherently needs an approximation of the transition probabilities. Therefore, as we have seen in Section 6 in, for the set of basis points we need to sample the subsequent states. This reduces FVI applicability as it might not be possible in practice. Additionally, both FVI and episodic VQ need careful hyper-parameter tuning to generate a satisfying policy, i.e., h and Z for FVI and ∆ for VQ. The big merit of LCNFQ is that it does not need any external intervention. Further, as in TAB1, LCNFQ succeeds to efficiently generate a better policy compared to FVI and VQ. LCNFQ has less sample complexity while at the same time produces policies that are more reliable and also has better expected reward, i.e. higher probability of satisfying the given property. This paper proposes LCNFQ, a method to train Q-function in a continuous-state MDP such that the ing traces satisfy a logical property. The proposed algorithm uses hybrid modes to automatically switch between neural nets when it is necessary. LCNFQ is successfully tested in a numerical example to verify its performance. e. s i+1 belongs to the smallest Borel set B such that P (B|s i, a i) = 1 (or in a discrete MDP, P (s i+1 |s i, a i) > 0). We might also denote ρ as s 0.. to emphasize that ρ starts from s 0.Definition A.2 (Stationary Policy) A stationary (randomized) policy Pol: S × A → is a mapping from each state s ∈ S, and action a ∈ A to the probability of taking action a in state s. A deterministic policy is a degenerate case of a randomized policy which outputs a single action at a given state, that is ∀s ∈ S, ∃a ∈ A, Pol (s, a) = 1.In an MDP M, we define a function R: S × A → R + 0 that denotes the immediate scalar bounded reward received by the agent from the environment after performing action a ∈ A in state s ∈ S.Definition A.3 (Expected (Infinite-Horizon) Discounted Reward) For a policy Pol on an MDP M, the expected discounted reward is defined as BID35: DISPLAYFORM0 where E Pol [·] denotes the expected value given that the agent follows policy Pol, γ ∈ is a discount factor and s 0,..., s n is the sequence of states generated by policy Pol up to time step n. Definition A.4 (Optimal Policy) Optimal policy Pol * is defined as follows: DISPLAYFORM1 where D is the set of all stationary deterministic policies over the state space S. The simplest way to solve an infinite-state MDP with RL is to discretise the state space and then to use the conventional methods in RL to find the optimal policy BID33. Although this method can work well for many problems, the ing discrete MDP is often inaccurate and may not capture the full dynamics of the original MDP. One might argue that by increasing the number of discrete states the latter problem can be resolved. However, the more states we have the more expensive and time-consuming our computations will be. Thus, MDP discretisation has to always deal with the trade off between accuracy and the curse of dimensionality. Let the MDP M be a finite-state MDP. Q-learning (QL), a sub-class of RL algorithms, is extensively used to find the optimal policy for a given finite-state MDP BID35. For each state s ∈ S and for any available action a ∈ A, QL assigns a quantitative value Q: S × A → R, which is initialized with an arbitrary and finite value for all state-action pairs. As the agent starts learning and receiving rewards, the Q-function is updated by the following rule when the agent takes action a at state s: DISPLAYFORM0 where Q(s, a) is the Q-value corresponding to state-action (s, a), 0 < µ ≤ 1 is called learning rate or step size, R(s, a) is the reward obtained for performing action a in state s, γ is the discount factor, and s is the state obtained after performing action a. Q-function for the rest of the state-action pairs remains unchanged. Under mild assumptions, for finite-state and finite-action spaces QL converges to a unique limit, as long as every state action pair is visited infinitely often BID40. Once QL converges, the optimal policy Pol *: S → A can be generated by selecting the action that yields the highest Q, i.e., Pol DISPLAYFORM1 where Pol * is the same optimal policy that can be generated via DP with Bellman operation. This means that when QL converges, we have DISPLAYFORM2 where s ∈ B is the agent new state after choosing action a at s such that P (B|s, a) = 1. Recall the QL update rule, in which the agent stores the Q-values for all possible state-action pairs. In the case when the MDP has a continuous state space it is not possible to directly use standard QL since it is practically infeasible to store Q(s, a) for every s ∈ S and a ∈ A. Thus, we have to turn to function approximators in order to approximate the Q-values of different state-action pairs of the Q-function. Neural Fitted Q-iteration (NFQ) BID26 is an algorithm that employs neural networks BID14 to approximate the Q-function, due to the ability of neural networks to generalize and exploit the set of samples. NFQ, is the core behind Google famous algorithm Deep Reinforcement Learning BID19.The update rule in can be directly implemented in NFQ. In order to do so, a loss function has to be introduced that measures the error between the current Q-value and the new value that has to be assigned to the current Q-value, namely DISPLAYFORM0 Over this error, common gradient descent techniques can be applied to adjust the weights of the neural network, so that the error is minimized. In classical QL, the Q-function is updated whenever a state-action pair is visited. In the continuous state-space case, we may update the approximation in the same way, i.e., update the neural net weights once a new state-action pair is visited. However, in practice, a large number of trainings might need to be carried out until an optimal or near optimal policy is found. This is due to the uncontrollable changes occurring in the Q-function approximation caused by unpredictable changes in the network weights when the weights are adjusted for one certain state-action pair BID25. More specifically, if at each iteration we only introduce a single sample point the training algorithm tries to adjust the weights of the neural network such that the loss function becomes minimum for that specific sample point. This might in some changes in the network weights such that the error between the network output and the previous output of sample points becomes large and failure to approximate the Q-function correctly. Therefore, we have to make sure that when we update the weights of the neural network, we explicitly introduce previous samples as well: this technique is called "experience replay" and detailed later. The core idea underlying NFQ is to store all previous experiences and then reuse this data every time the neural Q-function is updated. NFQ can be seen as a batch learning method in which there exists a training set that is repeatedly used to train the agent. In this sense NFQ is an offline algorithm as experience gathering and learning happens separately. We would like to emphasize that neural-net-based algorithms exploit the positive effects of generalization in approximation while at the same time avoid the negative effects of disturbing previously learned experiences when the network properly learns BID26. The positive effect of generalization is that the learning algorithm requires less experience and the learning process is highly data efficient. As stated earlier, many existing RL algorithms, e.g. QL, assume a finite state space, which means that they are not directly applicable to continuous state-space MDPs. Therefore, if classical RL is employed to solve an infinite-state MDP, the state space has to be discretized first and then the new discrete version of the problem has to be tackled. The discretization can be done manually over the state space. However, one of the most appealing features of RL is its autonomy. In other words, RL is able to achieve its goal, defined by the reward function, with minimum supervision from a human. Therefore, the state space discretization should be performed as part of the learning task, instead of being fixed at the start of the learning process. Nearest neighbor vector quantization is a method for discretizing the state space into a set of disjoint regions BID10. The Voronoi Quantizer (VQ) BID15, a nearest neighbor quantizer, maps the state space S onto a finite set of disjoint regions called Voronoi cells. The set of centroids of these cells is denoted by C = {c i} m i=1, c i ∈ S, where m is the number of the cells. Therefore, designing a nearest neighbor vector quantizer boils down to coming up with the set C. With C, we are able to use QL and find an approximation of the optimal policy for a continuous-state space MDP. The details of how the set of centroids C is generated as part of the learning task in discussed in the body of the paper. Finally, this section introduces Fitted Value Iteration (FVI) for continuous-state numerical dynamic programming using a function approximator BID9. In standard value iteration the goal is to find a mapping (called value function) from the state space to R such that it can lead the agent to find the optimal policy. The value function in our setup is when Pol is the optimal policy, i.e. U Pol *. In continuous state spaces, no analytical representation of the value function is in general available. Thus, an approximation can be obtained numerically through approximate value iteration, which involves approximately iterating the Bellman operator T on some initial value function BID33. FVI is explored more in the paper. It has been proven that FVI is stable and converging when the approximation operator is non-expansive BID9. The operator L is said to be non-expansive if:

Represent this text to find a much much longer article about it with say hundreds-thousands of words

As safety is becoming a critical notion in machine learning we believe that this work can act as a foundation for a number of research directions such as safety-aware learning algorithms.

We view exploration in RL as a problem of matching a marginal distribution over states.


Question: Represent the scientific article to find a short TLDR

Question: We introduce two approaches for conducting efficient Bayesian inference in stochastic simulators containing nested stochastic sub-procedures, i.e., internal procedures for which the density cannot be calculated directly such as rejection sampling loops. The ing class of simulators are used extensively throughout the sciences and can be interpreted as probabilistic generative models. However, drawing inferences from them poses a substantial challenge due to the inability to evaluate even their unnormalised density, preventing the use of many standard inference procedures like Markov Chain Monte Carlo (MCMC). To address this, we introduce inference algorithms based on a two-step approach that first approximates the conditional densities of the individual sub-procedures, before using these approximations to run MCMC methods on the full program. Because the sub-procedures can be dealt with separately and are lower-dimensional than that of the overall problem, this two-step process allows them to be isolated and thus be tractably dealt with, without placing restrictions on the overall dimensionality of the problem. We demonstrate the utility of our approach on a simple, artificially constructed simulator. Stochastic simulators are used in a myriad of scientific and industrial settings, such as epidemiology , physics , engineering and climate modelling . They can be complex and highdimensional, often incorporating domain-specific expertise accumulated over many years of research and development. As shown by the probabilistic programming (; van de ; and approximate Bayesian computation (ABC) (Csilléry et al., 2010;) literatures, these simulators can be interpreted as probabilistic generative models, implicitly defining a probability distribution over their internal variables and outputs. As such, they form valid targets for drawing Bayesian inferences. In particular, by constraining selected internal variables or outputs to take on specific values, we implicitly define a conditional distribution, or posterior, over the remaining variables. This effectively allows us, amongst other things, to run the simulator in "reverse", fixing the outputs to some observed values and figuring out what parameter values might have led to them. For example, given a simulator for visual scenes, we can run inference on the simulator with an observed image to predict what objects are present in the scene . Though recent advances in probabilistic programming systems (; ; ;) have provided convenient mechanisms for encoding, reasoning about, and constructing inference algorithms for such simulators, performing the necessary inference is still often extremely challenging, particularly for complex or high-dimensional problems. In this paper, we consider a scenario where this inference is particularly challenging to perform: when the simulator makes calls to nested stochastic sub-procedures (NSSPs). These NSSPs can take several different forms, such as internal rejection sampling loops, separate inference procedures, external sub-simulators we have no control over, or even realworld experiments. Their unifying common feature is that the density of their outputs cannot be evaluated up to an input-independent normalising constant in closed form. This, in turn, means the normalised density of the overall simulator cannot be evaluated, preventing one from using most common inference methods, including almost all Markov chain Monte Carlo (MCMC) and variational methods. Though some inference methods can still be applied in these scenarios, such as nested importance sampling , these tend to scale very poorly in the dimensionality and often even have fundamentally slower convergence rates than standard Monte Carlo approaches . To address this issue, we introduce two new approaches for performing inference in such models. Both are based around approximating the individual NSSPs. The first approach directly approximates the conditional density of the NSSP outputs using an amortized inference artefact. This then forms a surrogate density for the NSSP, which, once trained, is used to replace it. While this first approach is generally applicable, our second approach focuses on the specific case where the unnormalized density of the NSSP can be evaluated in isolation (such as a nested probabilistic program or rejection sampling loop), but its normalizing constant depends on the NSSP inputs. Here, we train a regressor to approximate the normalising constant of the NSSP as a function of its inputs. Once learnt, this allows the NSSP to be collapsed into the outer program: the ratio of the known unnormalised density and the approximated normalizing constant can be directly used as a factor in the overall density. Both approaches lead to an approximate version of the overall unnormalised density, which can then be used as a target for conventional inference methods like MCMC and variational inference. Because these approximations can be calculated separately for each NSSP, this allows them to scale to higher dimensional overall simulators far more gracefully than existing approaches, opening the door to tractably running inference for more complex problems. Furthermore, once trained, the approximations can be reused for different datasets and configurations of the outer simulator, thereby helping amortise the cost of running multiple different inferences for no extra cost. The approaches themselves are also amenable to automation, making them suitable candidates for PPS inference engines. We now introduce our two approaches for approximating NSSPs and show how these, in turn, produce efficient inference algorithms for the overall simulator. Both our approaches involve the gradient-based learning of a neural-network-based amortised approximation for each NSSP that takes in the NSSP inputs and either returns and approximation of the density of the outputs (method 1) or the normalizing constant (method 2). For any simulator or program, we can define the program density over valid program traces x 1:nx as (, Section 4.3.2): where n x is the length of the trace; each f a j (x j |φ j) represents the density of the j th random draw, which is made at location a j and takes in parameters φ j; and n y is a number of "observations", each of which factor the trace density by g b k (y k |ψ k), where b k is the location of this observation statement, y k is the observed value, and ψ k are parameters of the factorization. Here all terms-i.e., x j, n x, a j, φ j, n y, b k, y k, and ψ k -may be random variables, but each is deterministically calculable from the trace x 1:nx (see Rainforth (2017, Section 4.3.2)) A NSSP can now be formally defined as a f a j (x j |φ j) term which cannot be directly evaluated exactly, but where for a given φ j either [Case A] we can draw samples from f a j (x j |φ j) directly and/or [Case B] f a j (x j |φ j) corresponds to the normalized density of a nested probabilistic program that we can draw approximate samples from by running an separate inference procedure. Many simulators contain such sampling procedures (; ; ; ; ;, and it is these simulators that we target with our inference schemes. We can denote the unnormalized density for a program containing NSSPs as where is a representation of the "forward" or "prior" program which ignores all conditioning statements; S r = {a 1, . . ., a n} represents the set of addresses that produce intractable densities; and we use P in a j (x j |φ j) to distinguish the NSSPs from tractable sampling terms. Both our methods are now based on replacing each of the P in a j (x j |φ j) with an approximation, for which we only need to consider the prior program. Once learned, these can then be used to construct a directly evaluable approximate target densityγ(x 1:nx) by replacing each P in a j (x j |φ j) in, then running an MCMC sampler onγ(x 1:nx). Our first method replaces each P in a j (x j |φ j) by an approximate surrogate q in a j (x j |φ j ; η a j): where κ = {η a j ; a j ∈ S r} are the surrogate parameters. As per existing amortized variational approaches (; ; ; ;), each q in a j (x j |φ j ; η a j) is taken as a variational distribution parametrized by deep neural network with weights η a j and which takes φ j as its input. Training of these networks is done by minimising the Kullback-Leibler (KL) divergence from P pr (x 1:nx) to q(x 1:nx ; κ) This minimization can be done using stochastic gradient descent where the updates for NSSP r ∈ S r use the following gradient estimate (see Appendix A) and the x n 1:nx can be shared such that the variational approximations for each r ∈ S r are made simultaneously. Carrying out these updates requires us to draw samples from P pr. If all of our NSSPs satisfy [Case A], this is not a problem as by assumption we can then draw samples from each P in a j (x j |φ j) and, in turn, samples from P pr. However, if our program contains NSSPs which only satisfy [Case B], this will require us to run a separate nested inference to generate the required x n j from the corresponding φ j. Though this may be potentially non-trivial, it is, crucially, far easier that running inference on the overall program: because P pr itself does not include any conditioning statements, generating these samples does not require inference to be run for the outer program. As such, each nested inference problem constitutes its own isolated problem which is far simpler than the overall inference problem. In other words, the role of sampling from P pr is only to generate example input-output pairs for each NSSP, with each surrogate than separately trained based on its local pairs. If all of our NSSPs satisfy [Case B], this implies that each has a known unnormalised density on its internal variables and unknown input-dependent normalizing constant that causes a double-intractability. If the functional form for all these normalizing constant where known, this would be sufficient to collapse all the NSSPs into the outer program and produce a directly evaluable density for the overall program. Our second method thus looks to learn regressors to predict the normalizing constants and thereby facilitate this. To formalize this, let us for now assume that the x j returned by each NSSP corresponds to its full set of internal random draws z, such that we can write where γ in a j (z j 1:n j x |φ j) can be evaluated directly (because it is itself an unnormalized probabilistic program density of the form), but I in a j (φ j) is an intractable normalization constant. If we now introduce a set of regressors R r (φ j ; τ r), ∀r ∈ S r (with parameters τ r) to approximate each I in r (φ j), we can approximate P pr as We can extend this approach to the case where by instead defining our reference measure in the space of X a:= {x j} j∈1:nx|a j / ∈Sr ∪ {z j 1:n j x} j∈1:nx|a j / ∈Sr and using the pre-image of the prior program density: P pr (X a). We can then run inference in this pre-image space and rely on the law of the unconscious statistician to ensure the samples produces are from the desired posterior (see e.g. Rainforth (2017, Section 4.3.2)). Learning the regressors R r (φ j ; τ r) is done in an analogous manner to method one. Namely we run the program forward to gather pairs {φ j,Î r (φ j)} for each NSSP, whereÎ r (φ j) is an unbiased approximation of I r (φ j), and then use this as a training dataset for learning the regressor. Specifically, for each NSSP we train a neural network regressor to minimize the expected squared error between R r (φ j ; τ r) andÎ r (φ j). As shown in Appendix B, with a sufficiently expressive neural network, this scheme ensures R r (φ j ; τ r) → I r (φ j) ∀φ j as the number of training pairs tends to infinity. In this section, we use a 60-d nested Guassian example, details of which are given in Appendix C. The model has been contrived so that we can analytically calculate the posterior means and therefore validate against ground truth values. Figure 2 demonstrates this for Method 1 ( for Method 2 are still being developed). We see that accurate inference was achieved for all but two of the marginal distributions (these were caused by issues in the stability of the neural network training, which is currently being investigated). Though still preliminary, these are very promising in that they demonstrate that we are able to perform effective inference in far higher dimensions that can be realistically achieved by importance sampling based approaches, which are the current standard in the field. For a given simulator, or program, we denote the proposal for the program as: where κ = {η a j ; a j ∈ S r} are the variational parameters. Using the information projection we construct the variational objective as follows: Thus, we define the gradients to use for the stochastic gradient ascent for each subproblem r ∈ S r as: where x n 1:nx iid ∼ P pr (x). During training we extract samples from each forward run and train each NSSP separately. To train our regressors, we use the L 2 -norm E R r (φ j ; τ r) −Î in r (φ j) 2 2 between our regressor R r (φ j ; τ r) and our approximations of the marginalÎ in r (φ j). We then learn parameters τ r so that it minimises this objective, ing in R r (φ j ; τ r) = I in r (φ j) in the limit of a large number of training samples if our neural network has sufficient capacity to exactly capture I in r (φ j). To see this note that where the second term does not depend on τ r and the first is minimized when when R r (φ j ; τ r) = I a j (φ j). Our objective is defined as: where the expectation over the inputs φ j is defined by running P pr forward and, if necessary, randomly selecting between the inputs that are passed to NSSP r if it is called more than once (this can further be Rao-Blackwellized by averaging over all the inputs passed to the NSSP instead of choosing between them). Thus, by running the simulator forward, collecting samples from the NSSPs generated from sampling the priors of each NSSP, we can make updates based on ∇ τ (R r (φ j ; τ) −Î r (φ j)) 2 to minimise L r. With this approach, we must be careful to avoid over and under-fitting. Once trained, we can run inference on the approximate, unnormalised, target: The approach outlined in Method 2 can be improved upon in the case where our nested sub-procedures are rejection samplers. For rejection samplers, we always have I(φ) = E[I(A(z, φ) = 1)] where A(z,, φ) = 1 indicates an accepted sample and the expectation is with respect to running a single iteration of the rejection sampling loop. The naive Monte Carlo estimate for I(φ), 1 N N n=1 I(A(z n, φ) = 1), is only unbiased, if N is independent of the z n. Typically, one would like to instead run the rejection sampler in the standard manner, by which we generate samples by running the sampler until a sample is accepted, at which point we have generated N a samples, where N a is not independent of the z n, such that the naive estimate is now biased. However, not doing this could, for example, return an estimatê (I)(φ) = 0 which could cause significant issues if not dealt with properly, while it may not be possible to generate both strictly positive and unbiased estimates for I(φ). This conundrum can be circumvented by instead trying to directly estimate 1/I(φ) and use this as the basis for the regressor. This is possible because rejection samplers have the property E[N a |φ] = 1/I(φ) as follows: (1 − I(φ)) n = 1 I(φ) Therefore, we learn our regressor R to go from φ j to E[N a |φ], exploiting the fact that N a is an unbiased estimate of the latter, and subsequently use P in a j (x j |φ j) ≈ γ in a j (x j |φ j)R a j (φ j ; τ a j) to construct the approximate objective. It is interesting to further note that such that it should also be useful to use this to develop pseudo-marginal samplers for such problems. We take the model of a high-dimensional multivariate Gaussian with unkown mean and sample certain dimensions such that they rely on Gaussian NSSPs. The purpose of such an example is to demonstrate the validity of our methodoly, as it is one o the few examples in which we analytically calculate the correct ground truth. The model takes the following form: and we can sample x 1:d sequentially from a Markov process. As the covariance matrix takes this structure we can use standard identities, as in Petersen et al., to analytically calculate the value of µ x, which is plotted in Figure 1. Histograms both the predicted and ground truth values are provided in Figure 2 for all 60 dimensions.

Represent the short summary for finding the corresponding scientific article (long)

We introduce two approaches for efficient and scalable inference in stochastic simulators for which the density cannot be evaluated directly due to, for example, rejection sampling loops.

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the scientific article to find a short TLDR

Question: While adversarial training can improve robust accuracy (against an adversary), it sometimes hurts standard accuracy (when there is no adversary). Previous work has studied this tradeoff between standard and robust accuracy, but only in the setting where no predictor performs well on both objectives in the infinite data limit. In this paper, we show that even when the optimal predictor with infinite data performs well on both objectives, a tradeoff can still manifest itself with finite data. Furthermore, since our construction is based on a convex learning problem, we rule out optimization concerns, thus laying bare a fundamental tension between robustness and generalization. Finally, we show that robust self-training mostly eliminates this tradeoff by leveraging unlabeled data. Neural networks trained using standard training have very low accuracies on perturbed inputs commonly referred to as adversarial examples BID11. Even though adversarial training BID3 BID5 can be effective at improving the accuracy on such examples (robust accuracy), these modified training methods decrease accuracy on natural unperturbed inputs (standard accuracy) BID5 BID18. Table 1 shows the discrepancy between standard and adversarial training on CIFAR-10. While adversarial training improves robust accuracy from 3.5% to 45.8%, standard accuracy drops from 95.2% to 87.3%.One explanation for a tradeoff is that the standard and robust objectives are fundamentally at conflict. Along these lines, Tsipras et al. BID13 and Zhang et al. BID18 construct learning problems where the perturbations can change the output of the Bayes estimator. Thus no predictor can achieve both optimal standard accuracy and robust accuracy even in the infinite data limit. However, we typically consider perturbations (such as imperceptible ∞ perturbations) which do not change the output of the Bayes estimator, so that a predictor with both optimal standard and high robust accuracy exists. Another explanation could be that the hypothesis class is not rich enough to contain predictors that have optimal standard and high robust accuracy, even if they exist BID8. However, Table 1 shows that adversarial training achieves 100% standard and robust accuracy on the training set, suggesting that the hypothesis class is expressive enough in practice. Having ruled out a conflict in the objectives and expressivity issues, Table 1 suggests that the tradeoff stems from the worse generalization of adversarial training either due to (i) the statistical properties of the robust objective or (ii) the dynamics of optimizing the robust objective on neural networks. In an attempt to disentangle optimization and statistics, we ask does the tradeoff indeed disappear if we rule out optimization issues? After all, from a statistical perspective, the robust objective adds information (constraints on the outputs of perturbations) which should intuitively aid generalization, similar to Lasso regression which enforces sparsity BID12.Contributions. We answer the above question negatively by constructing a learning problem with a convex loss where adversarial training hurts generalization even when the optimal predictor has both optimal standard and robust accuracy. Convexity rules out optimization issues, revealing a fundamental statistical explanation for why adversarial training requires more samples to obtain high standard accuracy. Furthermore, we show that we can eliminate the tradeoff in our constructed problem using the recently-proposed robust self-training BID14 BID0 BID7 BID17 on additional unlabeled data. In an attempt to understand how predictive this example is of practice, we subsample CIFAR-10 and visualize trends in the performance of standard and adversarially trained models with varying training sample sizes. We observe that the gap between the accuracies of standard and adversarial training decreases with larger sample size, mirroring the trends observed in our constructed problem. Recent from BID0 show that, similarly to our constructed setting, robust self-training also helps to mitigate the trade-off in CIFAR-10.Standard vs. robust generalization. Recent work BID10 BID15 BID4 BID6 has focused on the sample complexity of learning a predictor that has high robust accuracy (robust generalization), a different objective. In contrast, we study the finite sample behavior of adversarially trained predictors on the standard learning objective (standard generalization), and show that adversarial training as a particular training procedure could require more samples to attain high standard accuracy. We construct a learning problem with the following properties. First, fitting the majority of the distribution is statistically easy-it can be done with a simple predictor. Second, perturbations of these majority points are low in probability and require complex predictors to be fit. These two ingredients cause standard estimators to perform better than their adversarially trained robust counterparts with a few samples. Standard training only fits the training points, which can be done with a simple estimator that generalizes well; adversarial training encourages fitting perturbations of the training points making the estimator complex and generalize poorly. We consider mapping x ∈ X ⊂ R to y ∈ R where (x, y) is a sample from the joint distribution P and conditional densities exist. We denote by P x the marginal distribution on X. We generate the data as y = f (x) + σv i where DISPLAYFORM0 ∼ N and f: X → R. For an example (x, y), we measure robustness of a predictor with respect to an invariance set B(x) that contains the set of inputs on which the predictor is expected to match the target y. The central premise of this work is that the optimal predictor is robust. In our construction, we let f be robust by enforcing the invariance property (see Appendix A) DISPLAYFORM1 Given training data consisting of n i.i.d. samples (x i,y i) ∼ P, our goal is to learn a predictor f ∈ F. We assume that the hypothesis class F contains f and consider the squared loss. Standard training simply minimizes the empirical risk over the training points. Robust training seeks to enforce invariance to perturbations of training points by penalizing the worst-case loss over the invariance set B(x i) with respect to target y i. We consider regularized estimation and obtain the following standard and robust (adversarially trained) estimators for sample size n: DISPLAYFORM2 DISPLAYFORM3 We construct a P and f such that both estimators above converge to f, but such that the error of the robust estimator f rob n is larger than that off std n for small sample size n. In our construction, we consider linear predictors as "simple" predictors that generalize well and staircase predictors as "complex" predictors that generalize poorly FIG1 ). Input distribution. In order to satisfy the property that a simple predictor fits most of the distribution, we define f to be linear on the set X line ⊆ X, where DISPLAYFORM0 for parameters δ ∈ and a positive integer s. Any predictor that fits points in X line will have low (but not optimal) standard error when δ is small. Perturbations. We now define the perturbations such that that fitting perturbations of the majority of the distribution requires complex predictors. We can obtain a staircase by flattening out the region around the points in X line locally FIG1 ). This motivates our construction where we treat points in X line as anchor points and the set X c line as local perturbations of these points: x ± for x ∈ X line. This is a simpler version of the commonly studied ∞ perturbations in computer vision. For a point that is not an anchor point, we define B(x) as the invariance set of the closest anchor point x. Formally, for some ∈ (0, DISPLAYFORM1 for some parameter m. Setting the slope as m = 1 makes f resemble a staircase. Such an f satisfies the invariance property that ensures that the optimal predictor for standard error is also robust. Note that f (x) = mx (a simple linear function) when restricted to x in X line. Note also that the invariance sets B(x) are disjoint. This is in contrast to the example in BID18, where any invariant function is also globally constant. Our construction allows a non-trivial robust and accurate estimator. We generate the output by adding Gaussian noise to the optimal predictor f, i.e., y = f (x)+σv i where v i DISPLAYFORM2 ∼ N. An illustration of our convex problem with slope m = 1, with size of the circles proportional to probability under the data distribution. The dashed blue line shows a simple linear predictor that has low test error but not robust to perturbations to nearby low-probability points, while the solid orange line shows the complex optimal predictor f that is both robust and accurate. (b): With small sample size (n = 40), any robust predictor that fits the sets B(x) is forced to be a staircase that generalizes poorly. (c): With large sample size (n = 25000), the training set contains all the points from Xline and the robust predictor is close to f by enforcing the right invariances. The standard predictor also has low error, but higher than the robust predictor. (d): An illustration of our convex problem when the slope m = 0. The optimal predictor f that is robust is a simple linear function. This setting sees no tradeoff for any sample size. We empirically validate the intuition that the staircase problem is sensitive to robust training by simulating training with various sample sizes and comparing the test MSE of the standard and robust estimators FORMULA2 and. We report final test errors here; trends in generalization gap (difference between train and test error) are nearly identical. See Appendix D for more details. FIG3 shows the difference in test errors of the two estimators. For each sample size n, we compare the standard and robust estimators by performing a grid search over regularization parameters λ that individually minimize the test MSE of each estimator. With few samples, most training samples are from X line and standard training learns a simple linear predictor that fits all of X line. On the other hand, robust estimators fit the low probability perturbations XAnother common approach to encoding invariances is data augmentation, where perturbations are sampled from B(x) and added to the dataset. Data augmentation is less demanding than adversarial training which minimizes loss on the worst-case point within the invariance set. We find that for our staircase example, an estimator trained even with the less demanding data augmentation sees a similar tradeoff with small training sets, due to increased complexity of the augmented estimator. Section 2.3 shows that the gap between the standard errors of robust and standard estimators decreases as training sample size increases. Moreover, if we obtained training points spanning X line, then the robust estimator (staircase) would also generalize well and have lower error than the standard estimator. Thus, a natural strategy to eliminate the tradeoff is to sample more training points. In fact, we do not need additional labels for the points on X line -a standard trained estimator fits points on X line with just a few labels, and can be used to generate labels on additional unlabeled points. Recent works have proposed robust self-training (RST) to leverage unlabeled data for robustness BID9 BID0 BID14 BID7 BID17. RST is a robust variant of the popular self-training algorithm for semi-supervised learning BID9, which uses a standard estimator trained on a few labels to generate psuedo-labels for unlabeled data as described above. See Appendix C for details on RST.For the staircase problem (m = 1), RST mostly eliminates the tradeoff and achieves similar test error to standard training (while also being robust, see Appendix C.2) as shown in FIG3. In our staircase problem from Section 2, robust estimators perform worse on the standard objective because these predictors are more complex, thereby generalizing poorly. Does this also explain the drop in standard accuracy we see for adversarially trained models on real datasets like CIFAR-10?. Difference between test errors (robust -standard) as a function of the # of training samples n. For each n, we choose the best regularization parameter λ for each of robust and standard training and plot the difference. Positive numbers show that the robust estimator has higher MSE than the standard estimator. (a) For the staircase problem with slope m = 1, we see that for small n, test loss of the robust estimator is larger. As n increases, the gap closes, and eventually the robust estimator has smaller MSE. (b) On subsampling CIFAR-10, we see that the gap between test errors (%) of standard and adversarially trained models decreases as the number of samples increases, just like the staircase construction in (a). Extrapolating, the gap should close as we have more samples. (c) Robust self-training (RST), using 1000 additional unlabeled points, achieves comparable test MSE to standard training (with the same amount of labeled data) and mostly eliminates the tradeoff seen in robust training. The shaded regions represent 1 STD.We subsample CIFAR-10 by various amounts to study the effect of sample size on the standard test errors of standard and robust models. To train a robust model, we use the adversarial training procedure from BID5 against ∞ perturbations of varying sizes (see FIG3 . The gap in the errors of the standard and adversarially trained models decreases as sample size increases, mirroring the trends in the staircase problem. Extrapolating the trends, more training data should eliminate the tradeoff in CIFAR-10. Similarly to the staircase example, BID0 showed that robust self-training with additional unlabeled data improves robust accuracy and standard accuracy in CIFAR-10. See Appendix C for more details. One of the key ingredients that causes the tradeoff in the staircase problem is the complexity of robust predictors. If we change our construction such that robust predictors are also simple, we see that adversarial training instead offers a regularization benefit. When m = 0, the optimal predictor (which is robust) is linear FIG1 ). We find that adversarial training has lower standard error by enforcing invariance on B(x) making the robust estimator less sensitive to target noise FIG6 ).Similarly, on MNIST, the adversarially trained model has lower test error than standard trained model. As we increase the sample size, both standard and adversarially trained models converge to obtain same small test error. We remark that our observation on MNIST is contrary to that reported in BID13, due to a different initialization that led to better optimization (see Appendix Section D.2). In this work, we shed some light on the counter-intuitive phenomenon where enforcing invariance respected by the optimal function could actually degrade performance. Being invariant could require complex predictors and consequently more samples to generalize well. Our experiments support that the tradeoff between robustness and accuracy observed in practice is indeed due to insufficient samples and additional unlabeled data is sufficient to mitigate this tradeoff. We show that the invariance condition (restated, FORMULA7) is a sufficient condition for the minimizers of the standard and robust objectives under P in the infinite data limit to be the same. DISPLAYFORM0 for all x ∈ X.Recall that y = f (x) + σv i where v i DISPLAYFORM1 if f is in the hypothesis class F, then f minimizes the standard objective for the square loss. If bothf std n andf rob n converge to the same Bayes optimal f as n → ∞, we say that the two estimatorsf std n andf rob n are consistent. In this section, we show that the invariance condition implies consistency off rob n andf std n.Intuitively, from, since f is invariant for all x in B(x), the maximum over B(x) in the robust objective is achieved by the unperturbed input x (and also achieved by any other element of B(x)). Hence the standard and robust loss of f are equal. For any other predictor, the robust loss upper bounds the standard loss, which in turn is an upper bound on the standard loss of f (since f is Bayes optimal). Therefore f also obtains optimal robust loss andf std n and f rob n are consistent and converge to f with infinite data. Formally, let be the square loss function, and the population loss be E (x,y)∼P [(f (x),y)]. In this section, all expectations are taken over the joint distribution P. Theorem 1. (Regression) Consider the minimizer of the standard population squared loss, f DISPLAYFORM2 2. Assuming holds, we have that for any f, E[maxx ∈B(x) (f (x), y)] ≥ E[maxx ∈B(x) (f * (x),y)], such that f * is also optimal for the robust population squared loss. Proof. Note that the optimal standard model is the Bayes estimator, such that f DISPLAYFORM3 where the first equality follows because f * is the Bayes estimator and the second equality is from BID6. Noting that for y) ], the theorem statement follows. DISPLAYFORM4 For the classification case, consistency requires label invariance, which is that argmax y p(y | x) = argmax y p(y |x) ∀x ∈ B(x), BID7 such that the adversary cannot change the label that achieves the maximum but can perturb the distribution. The optimal standard classifier here is the Bayes optimal classifier f c = argmax y p(y | x). Assuming that f c = argmax y p(y | x) is in F, then consistency follows by essentially the same argument as in the regression case. Proof. Replacing f with f c and (f (x), y) with the zero-one loss 1{argmax j f (x) j = y} in the proof of Theorem 1 gives the . In our staircase problem, from, we assume that the target y is generated as follows: y = f (x)+σv i where v i DISPLAYFORM5 ∼ N, we see that the points within an invariance sets B(x) have the same target distribution (target distribution invariance). DISPLAYFORM6 for all x ∈ X.The target invariance condition above implies consistency in both the regression and classification case. Distribution of X. We focus on a 1-dimensional regression case. Let s be the total number of "stairs" in the staircase problem. Let s 0 ≤ s be the number of stairs that have a large weight in the data distribution. Define δ ∈ to be the probability of sampling a perturbation point, i.e. x ∈ X c line, which we will choose to be close to zero. The size of the perturbations is ∈ [0, 1 2), which is bounded by 1 2 so that x± = x, for any x ∈ X line. The standard deviation of the noise in the targets is σ > 0. Finally, m ∈ is a parameter controlling the slope of the points in X line.Let w ∈ ∆ s be a distribution over X line where ∆ s is the probability simplex of dimension s. We define the data distribution with the following generative process for one sample x. First, sample a point i from X line according to the categorical distribution described by w, such that i ∼ Categorical(w). Second, sample x by perturbing i with probability δ such that DISPLAYFORM0 Note that this is just a formalization of the distribution described in Section 2. The sampled x is in X line with probability 1 − δ and X c line with probability δ, where we choose δ to be small. In addition, in order to exaggerate the difference between robust and standard estimators for small sample sizes, we set w such that the first s 0 stairs have the majority of probability mass. To achieve this, we set the unnormalized probabilities of w asŵ j = 1/s 0 j < s 0 0.01 j ≥ s 0 and define w by normalizing w =ŵ/ jŵ j. For our examples, we fix s 0 = 5. In general, even though we can increase s to create versions of our example with more stairs, s 0 is fixed to highlight the bad extrapolation behavior of the robust estimator. 2 ), where x rounds x to the nearest integer. The invariance sets are B(x) = {x −, x, x +}. We define the distribution such that for any x, all points in B(x) have the same mean target value m x. See FIG1 for an illustration. Note that B(x) is defined such that holds, since for any x 1,x 2 ∈ B(x), x 1 = x 2 and thus p(y | x 1) = p(y | x 2). The conditional distributions are defined since p(x) > 0 for anyx ∈ B(x). Our hypothesis class is the family of cubic B-splines as defined in BID2. Cubic B-splines are piecewise cubic functions, where the endpoints of each cubic function are called the knots. In our example, we fix the knots to be τ = [−,0,,...,(s−1)−,s−1,(s−1)+ ], which places a knot on every point on the support of X. This ensures that the family is expressive enough to include f, which is any function in F which satisfies f (x) = m x for all x in X. Cubic B-splines can be viewed as a kernel method with kernel feature map Φ: X → R 3s+2, where s is the number of stairs in the example. For some regularization parameter λ ≥ 0 we optimize with the penalized smoothing spline loss function over parameters θ, DISPLAYFORM0 where Ω i,j = Φ (t) i Φ (t) j dt measures smoothness in terms of the second derivative. With respect to the regularized objectives BID1 and, the norm regularizer is f 2 = θ T Ωθ. We implement the optimization of the standard and robust objectives using the basis described in BID2. The regularization penalty matrix Ω computes second-order finite differences of the parameters θ. Suppose we have n samples of training inputs X = {x 1,...,x n} and targets y = {y 1,...,y n} drawn from P. The standard spline objective solves the linear system DISPLAYFORM1 where the i-th row of Φ(X) ∈ R n×(3s+2) is Φ(x i). The standard estimator is thenf std n (x) = Φ(x) Tθ std. We solve the robust objective directly as a pointwise maximum of squared losses over the invariance sets (which is still convex) using CVXPY BID1. To construct an example where robustness hurts generalization, the main parameters needed are that the slope m is large and that the probability δ of drawing samples from perturbation points X c line is small. When slope m is large, the complexity of the true function increases such that good generalization requires more samples. A small δ ensures that a low-norm linear solution has low test error. This example is insensitive to whether there is label noise, meaning that σ = 0 is sufficient to observe that robustness hurts generalization. If m ≈ 0, then the complexity of the true function is low and we observe that robustness helps generalization. In contrast, this example relies on the fact that there is label noise (σ > 0) so that the noise-cancelling effect of robust training improves generalization. In the absence of noise, robustness neither hurts nor helps generalization since both the robust and standard estimators converge to the true function (f * (x) = 0) with only one sample. We show plots for a variety of quantities against number of samples n. For each n, we pick the best regularization parameter λ with respect to standard test MSE individually for robust and standard training. in the m = 1 (robustness hurts) and m = 0 (robustness helps) cases, with all the same parameters as before. In both cases, the test MSE and generalization gap (difference between training MSE and test MSE) are almost identical due to robust and standard training having similar training errors. In the m = 1 case where robustness hurts FIG10 ), robust training finds higher norm estimators for all sample sizes. With enough samples, standard training begins to increase the norm of its solution as it starts to converge to the true function (which is complex) and the robust train MSE starts to drop accordingly. In the m = 0 case where robustness helps (Figure 7), the optimal predictor is the line f (x) = 0, which has 0 norm. The robust estimator has consistently low norm. With small sample size, the standard estimator has low norm but has high test MSE. This happens when the standard estimator is close to linear (has low norm), but the estimator has the wrong slope, causing high test MSE. However, in the infinite data limit, both standard and robust estimators converge to the optimal solution. We describe the robust self-training procedure, which performs robust training on a dataset augmented with unlabeled data. The targets for the unlabeled data are generated from a standard estimator trained on the labeled training data. Since the standard estimator has good standard generalization, the generated targets for the unlabeled data have low error on expectation. Robust training on the augmented dataset seeks to improve both the standard and robust test error of robust training (over just the labeled training data). Intuitively, robust self-training achieves these gains by mimicking the standard estimator on more of the data distribution (by using unlabeled data) while also optimizing the robust objective. In robust self-training, we are given n samples of training inputs X = {x 1,...,x n} and targets y = {y 1,...,y n} drawn from P. Suppose that we have additional m unlabeled samples X u drawn from P x. Robust self-training uses the following steps for a given regularization λ:1. Compute the standard estimatorf std n on the labeled data (X, y) with regularization parameter λ.2. Generate pseudo-targets y u =f std n (X u) by evaluating the standard estimator obtained above on the unlabeled data X u. 3. Construct an augmented dataset X aug = X ∪ X u, y aug = y ∪ y u.4. Return a robust estimatorf rob n with the augmented dataset (X aug, y aug) as training data. We present relevant from the recent work of BID0 on robust self-training applied on CIFAR-10 augmented with unlabeled data in TAB2. The procedure employed in BID0 is identical to the procedure describe above, using a modified version of adversarial training (TRADES) BID18 as the robust estimator. In Section 2.4, we show that if we have access to additional unlabeled samples from the data distribution, robust self-training (RST) can mitigate the tradeoff in standard error between robust and standard estimators. It is important that we do not sacrifice robustness in order to have better standard error. FIG7 shows that in the case where robustness hurts generalization in our convex construction (m = 1), RST improves over robust training not only in standard test error BID5. The models are trained for 200 epochs using minibatched gradient descent with momentum, such that 100% standard training accuracy is achieved for both standard and adversarial models in all cases and > 98% adversarial training accuracy is achieved by adversarially trained models in most cases. We did not include reuslts for subsampling factors greater than 50, since the test accuracies are very low (20-50%). However, we note that for very small sample sizes (subsampling factor 500), the robust estimator can have slightly better test accuracy than the standard estimator. While this behavior is not captured by our example, we focus on capturing the observation that standard and robust test errors converge with more samples. The MNIST dataset consists of 60000 labeled examples of digits. We sub-sample the dataset by factors of {1, 2, 5, 8, 10, 20, 40, 50, 80, 200, 500} and report for a small 3-layer CNN averaged over 2 trials for each sub-sample factor. All models are trained for 200 epochs and achieve 100% standard training accuracy in all cases. The adversarial models achieve > 99% adversarial training accuracy in all cases. We train the adversarial models under the ∞ attack model with PGD adversarial training and = 0.3. For computing the max in each training step, we use 40 steps of PGD, with step size 0.01 (the parameters used in BID5). We use the Adam optimizer. The final robust test accuracy when training with the full training set was 91%.Initialization and trade-off for MNIST. We note here that the tradeoff for adversarial training reported in BID13 is because the adversarially trained model hasn't converged (even after a large number of epochs). Using the Xavier initialization, we get faster convergence with adversarial training and see no drop in clean accuracy at the same level of robust accuracy. Interestingly, standard training is not affected by initialization, while adversarial training is dramatically affected. Number of labeled samples Figure 7. Plots as number of samples varies for the case where robustness helps (m = 0). For each n, we pick the best regularization parameter λ with respect to standard test MSE individually for robust and standard training. (a),(b) The robust estimator has lower test MSE, and the gap shrinks with more samples. Note that the trend in test MSE is almost identical to generalization gap. (c) The robust estimator has consistent norm throughout due to the noise-cancelling behavior of optimizing the robust objective. While the standard estimator has low norm for small samples, it has high test MSE due to finding a low norm (close to linear) solution with the wrong slope.

Represent the TLDR for finding the corresponding scientific article

Even if there is no tradeoff in the infinite data limit, adversarial training can have worse standard accuracy even in a convex problem.