Question: Represent the scientific article to find a short TLDR

Question: Recently, there has been a surge in interest in safe and robust techniques within reinforcement learning (RL). Current notions of risk in RL fail to capture the potential for systemic failures such as abrupt stoppages from system failures or surpassing of safety thresholds and the appropriate responsive controls in such instances. We propose a novel approach to fault-tolerance within RL in which the controller learns a policy can cope with adversarial attacks and random stoppages that lead to failures of the system subcomponents. The of the paper also cover fault-tolerant (FT) control so that the controller learns to avoid states that carry risk of system failures. By demonstrating that the class of problems is represented by a variant of SGs, we prove the existence of a solution which is a unique fixed point equilibrium of the game and characterise the optimal controller behaviour. We then introduce a value function approximation algorithm that converges to the solution through simulation in unknown environments. Reinforcement learning (RL) provides the promise of adaptive agents being able to discover solutions merely through repeated interaction with their environment. RL has been deployed in a number of real-world settings in which, using RL, an adaptive agent learns to perform complex tasks, often in environments shared by human beings. Large scale factory industrial applications, traffic light control , robotics and autonomous vehicles are notable examples of settings to which RL methods have been applied. Numerous automated systems are however, susceptible to failures and unanticipated outcomes. Moreover, many real-world systems amenable to RL suffer the potential for random stoppages and abrupt failures; actuator faults, failing mechanical system components, sensor failures are few such examples. In these settings, executing preprogrammed behaviours or policies that have been trained in idealised simulated environments can prove vastly inadequate for the task of ensuring the safe execution of tasks. Consequently, in the presence of such occurrences, the deployment of RL agents introduces a risk of catastrophic outcomes whenever the agent is required to act so as to avoid adverse outcomes in unseen conditions. The important question of how to control the system in a way that is both robust against systemic faults and, minimises the risk of faults or damage therefore arises. In response to the need to produce RL algorithms that execute tasks with safety guarantees, a significant amount of focus has recently been placed on safe execution, robust control and riskminimisation (Garcıa and Fernández, 2015). Examples include H ∞ control , coherent risk, conditional value at risk . In general, these methods introduce an objective 1 defined with an expectation measure that either penalises actions that lead to greater uncertainty or embeds a more pessimistic view of the world (for example, by biasing the transition predictions towards less desirable states). In both cases, the ing policies act more cautiously over the horizon of the problem as compared to policies trained with a standard objective function. Despite the recent focus on safe methods within RL, the question of how to train an RL agent that can cope with random failures remains unaddressed. In particular, at present the question of how to produce an RL policy that can cope with an abrupt failure of some system subcomponent has received no systematic treatment. Similarly, the task of addressing how to produce RL policies that account for the risk of states in which such failures occur has not been addressed. In this paper, we for the first time produce a method that learns optimal policies in response to random and adversarial systems attacks that lead to stoppages of system (sub)components that may produce adverse events. Our method works by introducing an adversary that seeks to determine a stopping criterion to stop the system at states that lead to the worst possible (overall) outcomes for the controller. Using a game-theoretic construction, we then show how a policy that is robust against adversarial attacks that lead to abrupt failure can be learned by an adaptive agent using an RL updating method. In particular, the introduction of an adversary that performs attacks at states that lead to worst outcomes generates experiences for the adaptive RL agent to learn a best-response policy against such scenarios. To tackle this problem, we construct a novel two-player stochastic game (SG) in which one of the players, the controller, is delegated the task of learning to modify the system dynamics through its actions that maximise its payoff and an adversary or'stopper' that enacts a strategy that stops the system in such a way that maximises the controller's costs. This produces a framework that finds optimal policies that are robust against stoppages at times that pose the greatest risk of catastrophe. The main contribution of the paper is to perform the first systematic treatment of the problem of robust control under worst-case failures. In particular, we perform a formal analysis of the game between the controller and the stopper. Our main are centered around a minimax proof that establishes the existence of a value of the game. This is necessary for simulating the stopping action to induce fault-tolerance. Although minimax proofs are well-known in game theory (; ;), replacing a player's action set with stopping rules necessitates a minimax proof (which now relies on a construction of open sets) which markedly differs to the standard methods within game theory. Additionally, crucial to our analysis is the characterisation of the adversary optimal stopping rule (Theorem 3). Our tackle optimal stopping problems (OSPs) under worst-case transitions. OSPs are a subclass of optimal stochastic control (OSC) problems in which the goal is to determine a criterion for stopping at a time that maximises some state-dependent payoff . The framework is developed through a series of theoretical : first, we establish the existence of a value of the game which characterises the payoff for the saddle point equilibrium (SPE). Second, we prove a contraction mapping property of a Bellman operator of the game and that the value is a unique fixed point of the operator. Third, we prove the existence and characterise the optimal stopping time. We then prove an equivalence between the game of control and stopping and worst-case OSPs and show that the fixed point solution of the game solves the OSP. Finally, using an approximate dynamic programming method, we develop a simulation-based iterative scheme that computes the optimal controls. The method applies in settings in which neither the system dynamics nor the reward function are known. Hence, the agent need only observe its realised rewards by interacting with the environment. At present, the coverage of FT within RL is limited. In RL is applied to tackle systems in which faults might occur and subsequently incur a large cost. Similarly, RL is applied to a problem in in which an RL method for Bayesian discrimination which is used to segment the state and action spaces. Unlike these methods in which infrequent faults from the environment generate negative feedback, our method introduces an adversary that performs the task of simulating high-cost stoppages (hence, modelling faults) that induce an FT trained policy. A relevant framework is a two-player optimal stopping game (Dynkin game) in which each player chooses one of two actions; to stop the game or continue . Dynkin games have generated a vast literature since the setting requires a markedly different analysis from standard SG theory. In the case with one stopper and one controller such as we are concerned with, the minimax proof requires a novel construction using open sets to cope with the stopping problem for the minimax . Presently, the study of optimal control that combines control and stopping is limited to a few studies e.g. . Similarly, games of control and stopping have been analysed in continuous-time (; ;). In these analyses, all aspects of the environment are known and in general, solving these problems requires computing analytic solutions to non-linear partial differential equations which are often analytically insoluble and whose solutions can only be approximated numerically at very low dimensions. Current iterative methods in OSPs (and approximated dynamic programming methods e.g. ) in unknown environments are restricted to risk-neutral settings -introducing a notion of risk (generated adversarially) adds considerable difficulty as it requires generalisation to an SG involving a controller and stopper which alters the proofs throughout. In particular, the solution concept is now an SG SPE, the existence of which must be established. As we show, our framework provides an iterative method of solving OSPs with worst-case transitions in unknown environments and hence, generalises existing OSP analyses to incorporate a notion of risk. The paper is organised as follows: we firstly give a formal description of the FT RL problem we tackle and the OSP with worst-case transitions and give a concrete example to illustrate an application of the problem. In Sec. 2, we introduce the underlying SG framework which we use within the main theoretical analysis which we perform in Sec. 3. Lastly, in Sec. 4, we develop an approximate dynamic programming approach that enables the optimal controls to be computed through simulation, followed by some concluding remarks. We now describe the main problem with which we are concerned that is, FT RL. We later prove an equivalence between the OSPs under worst-case transitions and the FT RL problem and characterise the solution of each problem. We concern ourselves with finding a control policy that copes with abrupt system stoppages and failures at the worst possible states. Unlike standard methods in RL and game theory that have fixed time horizons (or purely random exit times) in the following, the process is stopped by a fictitious adversary that uses a stopping strategy or rule to decide when to stop given its state observations. In order to generate an FT control, we simulate the adversary's action whilst the controller determines its optimal policy. This as we show, induces a form of control that is an FT best-response control. A formal description is as follows: an agent exercises actions that influence the sequence of states visited by the system. At each state, the agent receives a reward which is dependent on the state and the chosen action. The agent's actions are selected by a policy π: S× A → -a map from the set of states S and the set of actions A to a probability. We assume that the action set is a discrete compact set and that the agent's policy π is drawn from a compact policy set Π. The horizon of the problem is T ∈ N × {∞}. However, at any given point τ S ≤ T the system may stop (randomly) and the problem terminates where τ S ∼ f ({0, . . ., T}) is a measurable, random exit time and f is some distribution on {0, . . ., T}. If after k ≤ T time steps the system stops, the agent incurs a cost of G(S k) and the process terminates. For any s ∈ S and for any π ∈ Π, the agent's performance function is given by: where a ∧ b:= min{a, b}, E is taken w.r.t. the transition function P. The performance function consists of a reward function R: S× A → R which quantifies the agent's immediate reward when the system transitions from one state to the next, a bequest function G: S → R which quantifies the penalty incurred by the agent when the system is stopped and γ ∈ [0, 1[, a discount factor. We assume R and G are bounded and measurable. The FT control problem which we tackle is one in which the controller acts both with concern for abrupt system failures and stoppages. In particular, the analysis is performed in sympathy with addressing the problem of how the controller should act in two scenarios -the first involves acting in environments that are susceptible to adversarial attacks or random stoppages in high costs states. Such situations are often produced in various real-world scenarios such as engine failures in autonomous vehicles, network power failures and digital (communication) networks attacks. The second scenario involves a controller that seeks to avoid system states that yield a high likelihood of systemic (subcomponent) failure. Examples of this case include an agent that seeks to avoid performing tasks that increase the risk of some system failure, for example increasing stress that in component failure or breakages within robotics. To produce a control that is robust in these scenarios, it is firstly necessary to determine a stopping rule that stops the system at states that incur the highest overall costs. Applying this stopping rule to the system subsequently induces a response by the controller that is robust against systemic faults at states in which stopping inflicts the greatest overall costs. This necessitates a formalism that combines an OSP to determine an optimal (adversarial) stopping rule and secondly, a RL problem. Hence, problem we consider is the following: where the minimisation is taken pointwise and V is a set of stochastic processes of the form v: Ω → T where T ⊆ {0, 1, 2 . . .} is a set of stopping times. Hereon, we employ the following shorthand R(s, a) ≡ R a s for any s ∈ S, a ∈ A. The dual objective consists of finding both a stopping rule that minimises J and an optimal policy that maximises J. By considering the tasks as being delegated to two individual players, the problem becomes an SG between a controller that seeks to maximise J by manipulating state visitations through its actions and an adversarial stopper that chooses a stopping rule to stop the process in order to minimise J. We later consider a setting in which neither player has up-front knowledge of the transition model or objective function but each only observes their realised rewards. The of this paper also tackle OSPs under a worst-case transitions -problems in which the goal is to find a stopping ruleτ under the adverse non-linear expectation E P:= min π∈Π E P,π s.th. Here, the agent seeks to find an optimal stopping time in a problem in which the system transitions according to an adversarial (worst-case) probability measure. To elucidate the ideas, we now provide a concrete practical example namely that of actuator failure within RL applications. Consider an adaptive learner, for example a robot that uses a set of actuators to perform actions. Given full operability of its set of actuators, the agent's actions are determined by a policy π: S ×A → which maps from the state space S and the set of actions A to a probability. In many systems, there exists some risk of actuator failure at which point the agent thereafter can affect the state transitions by operating only a subset of its actuators. In this instance, the agent's can only execute actions drawn from a subset of its action spaceÂ ⊂ A and hence, the agent is now restricted to policies of the form π partial: S ×Â → -thereafter its expected return is given by the value function V π partial (this plays the role of the bequest function G in). In order to perform robustly against actuator failure, it is therefore necessary to consider a set of stopping times T ⊆ {0, 1, 2, . . .} and a stopping criterionτ: Ω → T which determines the worst states for the agent's functionality to be impaired so that it can only use some subset of its set of actuators. The problem involves finding a pair (τ,π) ∈ V × Π -a stopping time and control policy s.th. where s:= s 0, a t ∼ π and Hence the role of the adversary is to determine and execute the stopping actionτ that leads to the greatest reduction in the controller's overall payoff. The controller in turn learns to execute the policyπ which involves playing a policyπ partial ∈ arg max V π partial after the adversary has executed its stopping action. The ing policyπ is hence robust against actuator failure at the worst possible states. Embedded within problem is an interdependence between the actions of the players -that is, the solution to the problem is jointly determined by the actions of both players and their responses to each other. The appropriate framework to tackle this problem is therefore an SG . In this setting, the state of the system is determined by a stochastic process {s t |t = 0, 1, 2, . . .} whose values are drawn from a state space S ⊆ R p for some p ∈ N. The state space is defined on a probability space (Ω, B, P), where Ω is the sample space, B is the set of events and P is a map from events to probabilities. We denote by F = (F n) n≥0 the filtration over (Ω, B, P) which is an increasing family of σ−algebras generated by the random variables s 1, s 2,.... We operate in a Hilbert space V of real-valued functions on L 2, i.e. a complete 2 vector space which we equip with a norm ·: is a probability measure. The problem occurs over a time interval {0, . . . K} where K ∈ N × {∞} is the time horizon. A stopping time is defined as a random variable τ: Ω → {0, . . ., K} for which {ω ∈ Ω|τ (ω) ≤ t} ∈ F t for any t ∈ {0, . . ., K} -this says that given the information generated by the state process, we can determine if the stopping criterion has occurred. An SG is an augmented Markov decision process which proceeds by two players tacking actions that jointly manipulate the transitions of a system over K rounds which may be infinite. At each round, the players receive some immediate reward or cost which is a function of the players' joint actions. The framework is zero-sum so that a reward for player I simultaneously represents a cost for player II. Formally, a two-player zero-sum SG is a 6−tuple S, A i∈{1,2}, P, R, γ where S = {s 1, s 2, . . ., s n} is a set of n ∈ N states, A i is an action set for each player i ∈ {1, 2}. The map P: S×A 1 ×A 2 ×S → is a Markov transition probability matrix i.e. P (s ; s, a 1, a 2) is the probability of the state s being the next state given the system is in state s and actions a 1 ∈ A 1 and a 2 ∈ A 2 are applied by player I and player II (resp.). The function R: S× A 1 × A 2 is the one-step reward for player I and represents one-step cost for player II when player I takes action a 1 ∈ A 1 and player II takes action a 2 ∈ A 2 and γ ∈ [0, 1[ is a discount factor. The goal of each player is to maximise its expected cumulative return -since the game is antagonistic, the total expected reward received by player I which we denote by J, represents a total expected cost for player II. Denote by Π i, the space of strategies for each player i ∈ {1, 2}. For SGs with Markovian transition dynamics, we can safely dispense with path dependencies in the space of strategies. 3 Consequently, w.log. we restrict ourselves to the class of behavioural strategies that depend only on the current state and round, namely Markov strategies, hence for each player i, the strategy space Π i consists of strategies of the form It is well-known that for SGs, an equilibrium exists in Markov strategies even when the opponent can draw from non-Markovian strategies . In SGs, it is usual to consider the case A 1 = A 2 so that the players' actions are drawn from the same set. We depart from this model and consider a game in which player II can choose a strategy which determines a time to stop the process contained within the set T ⊆ {0, 1, 2, . . .} which consists of F− measurable stopping times. In this setting, player I can manipulate the system dynamics by taking actions drawn from A 1 (we hereon use A) and at each point, player II can decide to intervene to stop the game. The value of the game exists if we can commute the max and min operators: We denote the value by J:= val and denote by (k,π) ∈ V × Π the pair that satisfies Jk,π ≡ J. The value, should it exist, is the minimum payoff each player can guarantee itself under the equilibrium strategy. In general, the functions val + [J] and val − [J] may not coincide. Should J exist, it constitutes an SPE of the game in which neither player can improve their payoff by playing some other control -an analogous concept to a Nash equilibrium for the case of two-player zero-sum games. Thus the central task to establish an equilibrium involves unambiguously assigning a value to the game, that is proving the existence of J. In this section, we present the key and perform the main analysis of the paper. Our first task is to prove the existence of a value of the game. This establishes a fixed or stable point which describes the equilibrium policies enacted by each player. Crucially, the equilibrium describes the maximum payoff that the controller can expect in an environment that is subject to adversarial attacks that stop the system or some subcomponent. Unlike standard SGs with two controllers, introducing a stopping criterion requires an alternative analysis in which i) an equilibrium with Markov strategies in which one of the players uses a stopping criterion is determined and ii) the stopping criterion is characterised. It is well-known that introducing a stopping action to one of the players alters the analysis of SGs the standard methods of which cannot be directly applied (c.f. Dynkin games ). Our second task is to perform an analysis that enables us to construct an approximate dynamic programming method. This enables the value function to be computed through simulation. This, as we show in Sec. 4, underpins a simulation-based scheme that is suitable for settings in which the transition model and reward function is a priori unknown. Lastly, we construct an equivalence between robust OSPs and games of control and stopping. We defer some of the proofs to the appendix. Our develop the theory of risk within RL to cover instances in which the agent has concern the process at a catastrophic system state. Consequently, we develop the theory of SGs to cover games of control and stopping when neither player has up-front environment knowledge. We prove an equivalence between robust OSPs and games of control and stopping and demonstrate how each problem can be solved in unknown environments. A central task is to prove that the Bellman operator for the game is a contraction mapping. Thereafter, we prove convergence to the unique value. Consider a Borel measurable function which is absolutely integrable w.r.t. the transition kernel ss, where P a ss ≡ P (s ; s, a) is the probability of the state s being the next state given the action a ∈ A and the current state is s. In this paper, we denote by (P J)(s):= S J[s]P a sds. We now introduce the operator of the game which is of central importance: The operator T enables the game to be broken down into a sequence of sub minimax problems. It will later play a crucial role in establishing a value iterative method for computing the value of the game. We now briefly discuss strategies. A player strategy is a map from the opponent's policy set to the player's own policy set. In general, in two player games the player who performs an action first employs the use of a strategy. Typically, this allows the player to increase its rewards since their action is now a function of the other player's later decisions. Markov controls use only information about the current state and duration of the game rather than using information about the opponent's decisions or the game history. Seemingly, limiting the analysis to Markov controls in the current game may restrict the abilities of the players to perform optimally. Our first however proves the existence of the value in Markov controls: Theorem 1 establishes the existence of the game which permits commuting the max and min operators of the objective. Crucially, the theorem secures the existence of an equilibrium pair (τ,π) ∈ V × Π, whereπ ∈ Π is the controller's optimal Markov policy when it faces adversarial attacks that stop the system. Additionally, Theorem 1 establishes the existence of a given by J, the computation of which, is the subject of the next section. We can now establish the optimal strategies for each player. To this end, we now define best-response strategies which shall be useful for further characterising the equilibrium: Definition 1. The set of best-response (BR) strategies for player I against the stopping time τ ∈ V (BR strategies for player II against the control policy π ∈ Π) is defined by: The question of computing the value of the game remains. To this end, we now prove that repeatedly applying T produces a sequence that converges to the value. In particular, the game has a fixed point property which is stated in the following: There exists a unique function J ∈ L 2 s.th. Theorem 2 establishes the existence of a fixed point of T and that the fixed point coincides with the value of the game. Crucially, it suggests that J can be computed by an iterative application of the Bellman operator which underpins a value iterative method. We study this aspect in Sec. 4 where we develop an iterative scheme for computing J. Definition 2. The pair (τ,π) ∈ V × Π is an SPE iff: An SPE therefore defines a strategic configuration in which both players play their BR strategies. With reference to the FT RL problem, an SPE describes a scenario in which the controller optimally responds against stoppages at the set of states that inflict the greatest costs to the controller. In particular, we will demonstrate thatπ ∈ Π is a BR to a system that undergoes adversarial attacks. Proposition 1. The pair (τ,π) ∈ V × Π consists of BR strategies and constitutes an SPE. By Prop. 1, when the pair (τ,π) is played, each player executes its BR strategy. The strategic response then induces FT behaviour by the controller. We now turn to the existence and characterising the optimal stopping time for player II. The following establishes its existence. Theorem 3. There exists an F-measurable stopping time: The theorem characterises and establishes the existence of the player II optimal stopping time which, when executed by the adversary, induces an FT control by the controller. Having shown the existence of the optimal stopping time τ, by Theorem 3 and Theorem 1, we find: Theorem 4. Letτ be the player II optimal stopping time defined in and let τ be the optimal stopping time for the robust OSP (c.f.) then τ =τ. Theorem 4 establishes an equivalence between the robust OSP and the SG of control and stopping hence, any method that computesτ for the SG yields a solution to the robust OSP. We now develop a simulation-based value-iterative scheme. We show that the method produces an iterative sequence that converges to the value of the game from which the optimal controls can be extracted. The method is suitable for environments in which the transition model and reward functions are not known to either player. The fixed point property of the game established in Theorem 2 immediately suggests a solution method for finding the value. In particular, we may seek to solve the fixed point equation (FPE) J = T J. Direct approaches at solving the FPE are not generally fruitful as closed solutions are typically unavailable. To compute the value function, we develop an iterative method that tunes weights of a set of basis functions {φ k : R p → R|k ∈ 1, 2, . . . D} to approximate J through simulated system trajectories and associated costs. Algorithms of this type were first introduced by Watkins as an approximate dynamic programming method and have since been augmented to cover various settings. Therefore the following can be considered as a generalised Q-learning algorithm for zero-sum controller stopper games. Let us denote by Φr:= D j=1 r(j)φ j an operator representation of the basis expansion. The algorithm is initialised with weight vector r 0 = (r 0,..., r 0 (P)) ∈ R d. Then as the trajectory {s t |t = 0, 1, 2, . . .} is simulated, the algorithm produces an updated series of vectors {r t |t = 0, 1, 2, . . .} by the update: Theorem 5 demonstrates that the method converges to an approximation of J. We provide a bound for the approximation error in terms of the basis choice. We define the function Q which the algorithm approximates by: We later show that Q serves to approximate the value J. In particular, we show that the algorithm generates a sequence of weights r n that converge to a vector r and that Φr, in turn approximates Q. To complete the connection, we provide a bound between the outcome of the game when the players use controls generated by the algorithm. We introduce our player II stopping criterion which now takes the form: Let us define a orthogonal projection Π and the function F by the following: ΠQ:= arg min We now state the main of the section: Theorem 5. r n converges to r where r is the unique solution: ΠF (Φr) = Φr. The following provide approximation bounds when employing the projection Π:, then the following hold: Hence the error bound in approximation of J is determined by the goodness of the projection. Theorem 5 and Theorem 6 thus enable the FT RL problem to be solved by way of simulating the behaviour of the environment and using the update rule to approximate the value function. Applying the stopping rule in, by Theorem 6 and Theorem 2, means the pair (τ,π) is generated where the policyπ approximates the policyπ which is FT against adversarial stoppages and faults. In this paper, we tackled the problem of fault-tolerance within RL in which the controller seeks to obtain a control that is robust against catastrophic failures. To formally characterise the optimal behaviour, we constructed a new discrete-time SG of control and stopping. We established the existence of an equilibrium value then, using a contraction mapping argument, showed that the game can be solved by iterative application of a Bellman operator and constructed an approximate dynamic programming algorithm so that the game can be solved by simulation. Assumption A.2. Ergodicity: i) Any invariant random variable of the state process is P −almost surely (P −a.s.) a constant. Assumption A.3. Markovian transition dynamics: the transition probability function P satisfies the following equality: Assumption A.4. The constituent functions {R, G} in J are square integrable: that is, R, G ∈ L 2 (µ). We begin the analysis with some preliminary lemmata and definitions which are useful for proving the main . Definition A.1. An operator T: V → V is said to be a contraction w.r.t a norm · if there exists a constant c ∈ [0, 1[ s.th for any V 1, V 2 ∈ V we have that: Definition A.2. An operator T : V → V is non-expansive if ∀V 1, V 2 ∈ V we have: Definition A.3. The residual of a vector V ∈ V w.r.t the operator T : V → V is: Lemma A.1. Define val + [f]:= min b∈B max a∈A f (a, b) and define val − [f]:= max a∈A min b∈B f (a, b), then for any b ∈ B we have that for any f, g ∈ L and for any c ∈ R >0: Lemma A.2. For any f, g, h ∈ L and for any c ∈ R >0 we have that: The following lemma, whose proof is deferred is a required for proving the contraction mapping property of the operator T. Lemma A.4. The probability transition kernel P is non-expansive, that is: The following estimates provide bounds on the value J which we use later in the development of the iterative algorithm. We defer the proof of the to the appendix. Proposition A.1. The operator T in is a contraction. Lemma A.5. Let T: V → V be a contraction mapping in · and let J be a fixed point so that T J = J then there exists a constant c ∈ [0, 1[ s.th: Lemma A.6. Let T 1 : V → V, T 2 : V → V be contraction mappings and suppose there exists vectors J 1, J 2 s.th T 1 J 1 = J 1 and T 2 J 2 = J 2 (i.e. J 1, J 2 are fixed points w.r.t T 1 and T 2 respectively) then ∃c 1, c 2 ∈ [0, 1[ s.th: Lemma A.7. The operator T satisfies the following: 2. (Constant shift) Let I(s) ≡ 1 be the unit function, then for any J ∈ L 2 and for any scalar α ∈ R, T satisfies T (J + αI)(s) = T J(s) + αI(s). Proof of Lemma A.1. We begin by noting the following inequality for any f: have that for all b ∈ V: From we can straightforwardly derive the fact that for any b ∈ V: (this can be seen by negating each of the functions in and using the properties of the max operator). Assume that for any b ∈ V the following inequality holds: Since holds for any b ∈ V and, by, we have in particular that whenever holds which gives the required . Lemma A.2 and Lemma A.3 are given without proof but can be straightforwardly checked. Proof of Lemma A.4. The proof is standard, we give the details for the sake of completion. Indeed, using the Tonelli-Fubini theorem and the iterated law of expectations, we have that: where we have used Jensen's inequality to generate the inequality. This completes the proof. Proof of Proposition A.1. We wish to prove that: Firstly, we observe that: 1[) and. The follows after applying Lemma A.2 and Lemma A.3. Proof of Lemma A.5. The proof follows almost immediately from the triangle inequality, indeed for any J ∈ L 2: where we have added and subtracted T J to produce the inequality. The then follows after inserting the definition of T (J). Proof of Lemma A.6. The proof follows directly from Lemma A.5. Indeed, we observe that for any J ∈ L 2 we have where we have added and subtracted J to produce the inequality. The then follows from Lemma A.5. Proof of Lemma A.7. Part 2 immediately follows from the properties of the max and min operators. It remains only to prove part 1. We seek to prove that for any s ∈ S, if J ≤J then We begin by firstly making the following observations: 1. For any x, y, h ∈ V x ≤ y =⇒ min{x, h} ≤ min{y, h}. 2. For any Assume that J ≤J, then we observe that: ≤ γ max where we have used in the penultimate line. The immediately follows after applying. Proof of Theorem 1. We begin by noting the following inequality holds: The inequality follows by noticing J k,π ≤ max We now observe that: where we have used the stationarity property and, in the limit m → ∞ and, in the last line we used the Fatou lemma. The constant c is given by c: Hence, we now find that Now since holds ∀π ∈ Π we find that: Lastly, applying min operator we observe that: It now remains to show the reverse inequality holds: Indeed, we observe that = min We now apply the min operator to both sides of which gives: After taking expectations, we find that: Now by Jensen's inequality and, using the stationarity of the state process (recall the expectation is taken under π) we have that: By standard arguments of dynamic programming, the value of the game with horizon n can be obtained from n iterations of the dynamic recursion; in particular, we have that: Inserting and into gives: where c(m):= Hence, we find that: we deduce the after noting that G(The proofs of the in Sec. 4 are constructed in a similar fashion that in (approximate dynamic programming). However, the analysis incorporates some important departures due to the need to accommodate the actions of two players that operate antagonistically. We now prove the first of the two of Sec. 4. Proof of Theorem 5. We firstly notice the construction ofτ given bŷ is sensible since we observe that min{t|G(s t) ≤ J } = min{t|G(s t) ≤ min{G(s t), Q (s t)} = min{t|G(s t) ≤ Q }. Step 1 Our first step is to prove the following bound: Proof. which is the required . Step 2 Our next task is to prove that the quantity Q is a fixed point of F and hence we can apply the operator F to achieve the approximation of the value. Proof. Using the definition of T (c.f. Step 3 We now prove that the operator ΠF is a contraction on Q, that is the following inequality holds: Proof. The proof follows straightforwardly by the properties of a projection mapping: Step 4 The is proven using the orthogonality of the (orthogonal) projection and by the Pythagorean theorem. Indeed, we have that: Proof. Φr − Q 2 = Φr − ΠQ 2 + ΠQ − Q Hence, we find that which is the required . Result 2 Proof. The proof by Jensen's inequality, stationarity and the non-expansive property of P. In particular, we have Inserting the definitions of Q andQ into then gives: It remains therefore to place a bound on the term Q −Q. We observe that by the triangle inequality and the fixed point properties of F on Q andF onQ we have ≤ γ Q − Φr + Q − Φr So that The then follows after substituting the of step 4. Let us now define the following quantity: Step 5 Proof. We now observe that s k can be described in terms of an inner product. Indeed, using the iterated law of expectations we have that Step 5 enables us to use classic arguments for approximate dynamic programming. In particular, following step 5, Theorem 6 follows directly from Theorem 2 in with only a minor adjustment in substituting the max operator with min.

Represent the TLDR for finding the corresponding scientific article

The paper tackles fault-tolerance under random and adversarial stoppages.

We introduce the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended to policy gradient methods.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We propose a novel framework to generate clean video frames from a single motion-blurred image. While a broad range of literature focuses on recovering a single image from a blurred image, in this work, we tackle a more challenging task i.e. video restoration from a blurred image. We formulate video restoration from a single blurred image as an inverse problem by setting clean image sequence and their respective motion as latent factors, and the blurred image as an observation. Our framework is based on an encoder-decoder structure with spatial transformer network modules to restore a video sequence and its underlying motion in an end-to-end manner. We design a loss function and regularizers with complementary properties to stabilize the training and analyze variant models of the proposed network. The effectiveness and transferability of our network are highlighted through a large set of experiments on two different types of datasets: camera rotation blurs generated from panorama scenes and dynamic motion blurs in high speed videos. Our code and models will be publicly available. Capturing an image is not an instant process; to capture enough photons, the photosensitive elements of a camera have to be exposed to light for a certain interval of time, called exposure time. Therefore, during this interval if an object is moving in the observed scene or the camera is undergoing an arbitrary motion, the ing image will contain a blurring artifact known as motion blur. In general, motion blur is an unwanted behaviour in vision applications e.g.image editing , visual SLAM (and 3D reconstruction , as it degrades the visual quality of images. To cope with this type of artifact, image deblurring aims to restore a sharp image from a blurred image. This problem is known to be ill-posed since the blur kernel used for deconvolution is generally assumed to be unknown. Earlier studies assume a uniform-blur over the image to simplify the estimation of the single deconvolution blur kernel used to remove the blur (; ;). Even though the methods deploy deblurring tasks with uniform-blur assumption, the assumption is often violated in practice. For instance, when the blur is caused by out-of-plane camera rotation, the blur pattern becomes spatially variant. Moreover, the problem is more complex when objects in a scene are moving i.e.dynamic blur. While previous literature focuses on recovering a sharp image from a blurred image, we tackle a more challenging task i.e.video restoration from a blurred image. Restoring the underlying image sequence of a blurred image requires both contents and motion prediction. We formulate video restoration from a blurred image as an inverse problem where a clean sequence of images and their motion as latent factors, and a blurred image as an observation. Some of previous deblurring approaches (; ; ; ;) also estimate the underlying motion in a blurred image, however, their goal remains in single frame restoration. proposed to extract video frames from a single motion-blurred image. Their approach is close to image translation model without inferring underlying motions between the latent frames. addressed this issue by estimating pixel level motion from a given blurred input. However, their model is still prone to sequential error propagation as frames are predicted in a sequential manner using a deblurred middle frame. In this paper, we propose a novel framework to generate a clean sequence of images from a single motion-blurred image. Our framework is based on a single encoder-decoder structure with Spatial Transformer Network modules (STN) and Local Warping layers (LW) to restore an image sequence and its underlying motion. Specifically, a single encoder is used to extract intermediate features which are passed to multiple decoders with predicted motion from STN and LW modules to generate a sequence of deblurred images. We evaluate our model on two types of motion blur. For rotation blur, which is caused by abrupt camera motion, we generated a synthetic dataset from panoramic images (J. Xiao & Torralba., 2012). For dynamic blur caused by fast moving objects in a scene, we used a high speed video dataset . The proposed model is evaluated on the panorama and the high speed video datasets under various motion patterns. Both the quantitative metrics and qualitative highlight that our method is more robust and performs favorably against the competing approach 1. For further investigation, we demonstrate the transferability of our model by cross-dataset evaluation. We also propose a simpler and lighter variation of our model guiding that our approach is flexible and can be easily extended to arbitrary number of frame prediction model with negligible performance trade-off. In short, our contributions are as follows. 1) We propose a novel unified architecture to restore clean video frames from a single motion-blurred image in an end-to-end manner. 2) Loss terms are designed to stably train the proposed network. 3) We perform thorough experiments to analyze the transferability and flexibility of the proposed architecture. 4) The performance of our model quantitatively and qualitatively performs favorably against the competing approach. Moreover due to flexibility of our model, we show that our approach is robust to heavy blurs where the previous approach fails. Image deblurring in general refers to the restoration of an image affected by blur. In this paper, we focus exclusively on the motion blur. Image deblurring is an ill-posed inverse problem when a blur kernel is unknown i.e.blind deconvolution problem, as different latent images can be transformed to a blurred image depending on its blur kernel. Early stage of deblurring studies (; ; ; ; ; ; ;) assume a single blur kernel that is applied to an image globally. The restoration of blur images is often modeled as a maximization problem of probabilistic models . To narrow down the ambiguity of the blur kernel estimation, natural image priors (; ; 2016;) are exploited. For instance, formulate the blur kernel estimation as a process to recover internal recurrence of image patches. 0 regularization , dark channel prior and extreme channel prior are also used to improve image deblurring. While single blur kernel estimation approaches are effective when blur kernels are shift-invariant, they fail when the blur is not spatially uniform. A non-uniform blur can be caused by camera rotation, depth variation or moving objects in a scene. To restore images affected by motion blur from pure rotations, use the geometric information of the camera motion as a prior to recover the non-uniform blur model. Recently, deep network based methods are proposed to handle general blur patterns without the uniform blur assumption. propose multi-scale deep networks with multi-scale loss that mimics coarse-to-fine approaches to restore sharp images under non-uniform blurred images. proposed a spatially variant neural networks to learn spatially variant kernels. proposed to extract a video sequence from a single motion-blurred image using multiple deep networks. They showed that deep networks can successfully generate an image sequence from a blurred image, however there remains a few limitations. Their proposed framework consists of multiple networks of which each network is specialized to predict a specific frame in a sequence. Each network is trained separately and sequentially starting from the middle frame and then adjacent frames taking previously predicted frames as inputs. As a , the non-middle frame prediction heavily relies on previously predicted frames including the middle frame itself, therefore when the middle frame is erroneous the error propagates across frames. proposed a two-step strategy to generate a video from a motion-blurred image using three complementary networks. They used video autoencoder to learn motion and frame generation from clean frames as a pretraining phase. Latter, they introduced a motion disentangle network to extract motion from blurred image. They also used independent deblurring network as their approach requires a clean middle frame generated from a blurred image in advance. Although their approach takes motion information into account, the approach generates frames sequentially starting from the middle frame to adjacent frames which in error propagation just as in. Unlike the previous works, our approach runs in an end-to-end manner within a single training stage without error propagation across frames. Collecting a large number of natural motion-blurred images is a daunting task. Hence, a common practice in computer vision research is to generate blurry images by combining a sequence of sharp images using various approaches ranging from simple averaging to learnable methods . The source of motion blur in an image can be generalized into two main categories: rapid camera motion (camera shake) and dynamic motion of objects in the scene. In this section, we briefly explain how we generate a blurry image dataset by considering each case individually. In order to generate a rotation blurred image dataset, we use the SUN360 panorama dataset (J. Xiao & Torralba., 2012). This dataset provides various panoramas with 360 • field of view. Hence, a virtual camera can be modeled to point at different orientations to represent the camera rotation in SO. Given a panorama P of size H × W, we developed a simple yet effective framework to generate blurred images. First, the panorama is projected onto a unit sphere by linearly mapping each pixel coordinate (x, y) ∈ P into spherical coordinates (θ, φ) with θ ∈ (0, 2π) and φ ∈ (−π/2, π/2). Then, a synthetic image can be captured via a virtual camera by re-projecting the 3D points on the sphere into an image plane as briefly discussed in and. Using this procedure we first capture an image by positioning the virtual camera at an arbitrary orientation. We call the image generated at this orientation initial image. Then, we rotate the camera by a random rotation matrix (with β = (β x, β y, β z) its Euler angle representation) and capture a second image at the new camera position called final image. We finally use a quaternion spherical linear interpolation technique (Slerp) The camera rotation angle is uniformly sampled from [−10 •, 10 •]. In order to generate a realistic blurred image, the number of intermediate images have to be adjusted automatically depending upon the rotation magnitude between the initial and final frames. Therefore, we use a simple linear relationship between the number of frames to be generated (n) and the rotation magnitude as follows: n = c + 1 3 β, where c is a constant and β is the magnitude of β. In this manner, we use 1000 panoramic images from which we generate 26, 000 training and 3, 200 test images of size 128 × 128px. In order to generate more realistic and generic (arbitrary camera motions and dynamic scene) blurred images, we take advantage of a GoPro high speed video dataset . This dataset provides 22 training and 11 test scenes, each scene containing frames of size 1280 × 720px. A blurry image is generated by averaging n consecutive frames . In our experiments, we fixed n = 7 and generated 20, 000 training and 2000 test images by randomly cropping images of size 256 × 256px. In this section, we describe our network structure and loss functions. To explicitly take the camera motion into consideration, our network is designed with spatial transformer networks (STNs) in an encoder-decoder structure (Fig. 2). Given a blurry image I b as an input, our model outputs {I 1, ..., I m−1, I m, I m+1, ..., I n}, where I m is the deblurred middle frame and {I j} n j=1 where j = m are the recovered non-middle frames. To ensure sharp image generation and stable training, the network is trained using three loss terms: multi-scale photometric loss, transformation consistency loss and penalty term. The middle frame I m is reconstructed using a U-net like network. The encoder contains five convolutional blocks, each block containing two layers of convolutions with spatial kernel size of 3 × 3 and stride size of 2 and 1, respectively. The feature maps are downsampled to half size and the number of channels is doubled after each convolutional block. The decoder network also contains five convolutional blocks to upsample features and to predict images at different scales. In each block, a feature is first upscaled using a transposed convolution (deconvolution) layer of kernel size 4×4 and a stride size of 2. The image predicted in the previous block is also upscaled in the same manner. The upsampled feature and its respective image are then concatenated channel-wise with the corresponding feature from the encoder (skip connection as shown in the Fig. 2a), then passed through five layers of convolutions with dense connections to output a feature, which will be used to predict an image at current block. In this manner, features and images are successively upsampled to predict a full scale middle frame. Along with the last feature map from the decoder, the predicted image is finally passed through a refining convolutional block. It contains seven dilated convolutional layers each with kernel size of 3×3 and different dilation constants. The purpose of this network is to further refine the predicted frame with contextual information by effectively enlarging the receptive field size of the network. The non-middle frames are reconstructed based on the encoded features of the middle frame via learned transformation by STN modules and local warping (LW) networks. First, the encoded middle frame feature U e ∈ R H×W ×C with width W, height H and C channels from the encoder is transformed into a non-middle frame feature using a feature transformer network (FTN). Second, the decoded middle frame image I d ∈ R H×W ×3 predicted from the corresponding middle frame decoder is transformed using an image transformer network (ITN). Third, the transformed feature and image are concatenated channel-wise and are passed through a decoder to predict a non-middle frame (Fig. 2b). We also input the middle frame feature into the non-middle frame decoder in order to guide the decoder to learn the spatial relation between the middle and the non-middle frame. The decoder network here is similar to the one used for predicting a middle frame in the previous section. These mechanisms are summarized in the following equations, where l = {1, ..., k} is an index for k feature levels (scales) and t is a subscript for transformed feature/image. I p is the predicted frame from a non-middle frame decoder D. Each non-middle frame is reconstructed by applying multi-scale transformer networks to the middle frame encoder. Given ground truth non-middle frames during training, our model learns the transformation parameters to be applied to the middle frame at different scales in order to output the desired non-middle frames. The fact that unique transformer networks are applied at each feature and image scale gives the model a capacity to learn various types of transformations, hence, making it robust to different blur patterns including large blurs. STNs in our model learn non-local transformations in a given motion-blurred image. In order to compensate for locally variant transformations, we designed a local warping network. This network is conditioned on the input feature like STN, however, instead of predicting global transformation parameters, it predicts pixel-wise displacement i.e.motion flow. Given an input feature U ∈ R H×W ×C, the local warping network outputs a motion flow of size H × W × 2. We used two convolutional layers of kernel size 3 × 3 for the network. By warping the input feature with the predicted motion flow, we obtain a locally transformed feature which is used as an input to the decoder (Fig. 2b). Given a blurry input, in order to generate video frames that are sharp both locally and globally, we trained our network with a multi-scale photometric loss between the images predicted by the decoder network and the ground truth image. A bilinear downsampling is used to resize the ground truth image to the corresponding predicted frame size at different scales. Let l denotes a scale level, k denotes total number of scales, {ŷ} k l=1 denotes a set of predicted images from the smallest size (ŷ 1) to the full scale (ŷ k), and {y} k l=1 represent a set of downsampled ground truth images where y k is a full scale ground truth image. For training a model predicting a sequence with n frames from a single blurry image, we compute multi-scale photometric loss as the term L mp in Eq., We use individual transformer networks for each feature level when predicting non-middle frames. This augments our model with the capacity to learn transformations at different levels making it robust to various blur patterns. However, we expect the transformations at different scales to be aligned for successfully reconstructing temporally consistent non-middle frames. Especially at the initial stages of the training where the transformer parameters are random, it is beneficial that our model understands the relationship between the transformations across different frame levels. In order to impose this notion into our model and facilitate a smooth training, we propose the transformation consistency loss. Let {θ} k l=1 be the set of predicted transformation parameters at different scales. The transformation consistency loss for predicting n − 1 non-middle frames can be defined as the term L tc in Eq., where |.| 2 is an 2 loss between the transformation parameters. Predicting multiple frames from a single blurry image can be problematic at times when the model fails to learn any type of transformation and simply replicates the middle frame predicition as nonmiddle frames. In order to remedy this issue, we design a penalty term to enforce diversity among generated images. This is accomplished by explicitly maximizing the sum of absolute difference (SAD) i.e. minimizing the negative SAD between a predicted frame and its time-symmetric (about the middle frame) ground truth frame. For example, when predicting seven frames {I 1, ..., I 4, ...., I 7}, we enforce the predicted image I 1 to be different content-wise from the ground truth image I 7 and vice versa. The penalty is imposed in a symmetric manner such that the model learns to be sensitive to smaller transformations close to the middle frame as well as larger transformations at the end frames. Given a predicted frameŷ i and the corresponding time-symmetric ground truth y n+1−i, the penalty term is computed as the term L p in Eq., where m is the middle frame index, n is the total number of frames. The final training loss function is defined as follows, where λ tc and λ p are weight coefficients for transformation consistency loss and penalty term, respectively. We used λ tc = 1e + 3 and λ p = 0.5. The task at hand has two main ambiguities. i. temporal shuffling and ii. reverse ordering. As explained in section 3, motion-blur is the of an averaging process and, restoring temporally consistent (no shuffling) sharp frame sequence from a given motion-blurred input is a non-trivial task as the averaging destroys the temporal order. mentions that photometric loss is not a sufficient constraint to make their network converge. Hence, they propose a pair-wise order invariant loss to train their network. also uses the same loss function to fine-tune the recurrent video decoder in their network. We find experimentally that a multi-scale photometric loss is a sufficient constraint to train our network. We further impose more constraints using other loss terms to improve performance (see ablation studies in Sec. 5.4.2). By design nature, our model allows motions to be learned in a symmetric manner (about the middle frame) with transformer networks close to the middle frame decoding smaller motions and those further from the middle frame decoding larger motions. This notion is enforced by transformation consistency loss and symmetric constraint term during training. The fact that our model is optimized in a joint manner allows frames to be reconstructed in a motion-guided sequence. Other than temporal shuffling, another issue is reverse ordering. Given a single motion-blurred input, recovering ground truth order is a highly ill-posed problem which is intractable since reversely ordered frames in the same motion-blurred image. Neither our work nor previous works (, ) are capable of predicting the right order. Hence, we evaluate frame reconstructions using both ground truth order and its reverse order, then report the higher metric in the experiment section. Our model is implemented and trained using pyTorch . We chose Adam as an optimizer with β 1 and β 2 fixed to 0.9 and 0.999, respectively. On our synthetic blur dataset, we train the model using images of size 128 × 128px and a mini-batch size of 8 to predict initial, middle and final frames. A mini-batch size of 4 and input size of 256 × 256px is used to predict sequences of frames when training on the high speed video dataset. In all experiments, we train our model for 80 epochs. We set the learning rate λ = 1e − 4 at the start of the training and decay it by half at epochs 40 and 60. All the training and test images are cropped from the original resolution images without resizing. In this section, we analyze the performance of our model qualitatively and quantitatively on both camera shake blurs generated from panoramic scenes and dynamic blurs obtained from averaging frames in high speed videos. We report test using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) metrics. To purely evaluate the quality of generated images without ordering estimation issue due to reverse ordering, we report the higher PSNR/SSIM metric of either ground truth order or reverse order of frames i.e.max{PSNR/SSIM( For dynamic blur, we compare our with 2 as they also use the same high speed video dataset . The averaged metrics obtained on the testing set are summarized in Table 1. The show that our model performs favorably against their model on both PSNR and SSIM on the high speed video dataset. While the middle frame prediction shows moderate performance increase, for the initial and final frame predictions, our model outperforms by a large margin. The performance gap between the middle frame and non-middle frames is relatively larger in than our method. This is due to sequential prediction in which makes non-middle frame prediction heavily dependent on the generated middle frame, ing in error propagation. As stated in , this limitation is particularly problematic when a heavy blur affects the input image since the middle frame prediction becomes less reliable. Our approach is Figure 3: Rotation blurred images generated from panorama scenes. The top row is ground truth frames and the bottom row is restored frames from the blurs. Input Figure 5: Partially blurred dynamic motion example. The man is blurred with dynamic motion while the is close to static. relatively robust to heavy blur as the proposed model generates frames independently from multiple decoders, therefore the error is not propagated (Fig. 7). We also evaluate our model on camera rotational blurs (Table 1 Panorama blur). In terms of blurriness, the panorama scenario is more challenging as we simulate broader range of blurs than the blurs in the high speed video dataset i.e.from sharp images to heavy blurred images. We observed visually better for the panorama scenario but lower quantitative number. One main reason is that panorama ground truths are relatively sharper while the high speed video contains blurry ground truth images due to dynamic motion and short exposure time. Therefore, the visual of panorama scenes from our model are sharp even though the quantitative performance is relatively lower. Please refer to Sec. 7.1 in the appendix for the quantitative on seven frame predictions. The qualitative for panoramic scenes and high speed videos show that our model can successfully restore multiple frames from a blurred input under various blur patterns. The restored frames from a blurred image in the panorama scenario (Fig. 3) are relatively sharper than dynamic blur case. As mentioned earlier, one of reasons is due to high quality ground truth image. In the high speed video scenario (Fig. 4), ground truth images have locally blurred contents due to fast dynamic motions. We compare our approach and previous method on relatively heavily blurred images from the high speed video dataset. As can be seen from Fig. 4, our method reconstructs contents consistently across frames and restores visually sharper videos compared to. In case of dynamic blurs where the motion of multiple objects and static objects are Figure 6: STN transformation visualization. The ground truth middle frame is transformed to non-middle frame using pure STN transformation. mixed as shown in Fig. 5, our model can generate video frames unravelling the underlying spatially varying motion. Failure cases happen for undersampled and heavily blurred inputs as can be seen from Fig. 7 and Fig. 9d. Please refer to Sec. 8 in the appendix for more qualitative . In order to visually analyze if the STN module in our model reasonably infers the blur motion, we apply the STN transformation to the middle frame and compare it with the ground truth non-middle frames. Let i be the index of the initial frame, m the middle frame, and f the final frame. The input image is obtained by averaging a set of frames {F j} f j=i. We can obtain the STN transformation parameters {θ} f j=1,j =m from the feature transformer network which transforms middle frame features to non-middle frame features. We apply the STN transformation T with the parameter θ j to middle frame F m to visualize whether the transformation implies valid motion information. Fig. 6 shows that the transformation spatially aligns contents of the middle frame to that of the non-middle frames. We report a cross-dataset panorama→high speed video evaluation to assess the generalization capability of our model. A model trained on the panoramic scenes is evaluated on high speed video test set (Table 2). Despite a performance degradation, our model trained on the panorama dataset performs on par with the competing approach trained on the high speed video dataset. The absence of dynamic motion on the panorama dataset, which is apparent in high speed videos, can be one contributing factor explaining the performance loss in addition to the domain gap e.g.image contents, sharpness, blurriness. Model size can be a bottleneck when the number of frames in a video to be predicted increases since our model uses multiple decoders and transformer networks. Hence, we experiment with a lighter model by replacing decoders and STN modules to weight shared layers. As opposed to using individual decoders for each non-middle frame, we use a single decoder i.e.weight shared decoders. For STN, we apply inverse transformations by assuming symmetric motion about the middle frame, therefore, reducing the number of transformer networks by half. We tested this light architecture on a model that predicts three (initial, middle and final) frames and it reduces the number of model parameters by 48%, yet yielding only a 0.4dB performance decrease on average (Table 2). The feature transformer network (FTN) at different levels of abstraction is the core part of our model for network convergence. Local warping (LW) layer also significantly improves the performance of our model. The best model performance is, yet, achieved with the three network components (FTN, LW, ITN) combined (Table 3). As mentioned earlier, the multi-scale photometric loss (PML) is a sufficient constraint to make our network converge during training. We also experimentally find that a model trained with transformation consistency loss (TCL) not only converges faster with smoother behavior but also gives a better performance during testing. The penalty term (PT) gives a marginal performance improvement when predicting fewer frames as photometric loss is already a sufficient constraint. In 3 frame prediction model, the penalty term improved performance marginally around 0.25dB while in 7 frame prediction model, it improved approximately 0.6dB. Penalty term enforces the model to consider subtle differences especially when the motion is small. We present a novel unified architecture that restores video frames from a single blurred image in an end-to-end manner. During training, feature and image transformer networks indirectly learn blur motions without motion supervision. The designed a loss function with regularizers enforce the model to consider subtle differences with fast loss convergence. We evaluate our model on the two datasets with rotation blurs and dynamic blurs and demonstrate qualitatively and quantitatively favorable performance against the competing approach. The cross-dataset evaluation demonstrates that our model can generalize even when the training and test set have significantly different blur patterns and domain gap. We additionally propose a lighter version of the model by weight-sharing of the decoders and STN modules under symmetric frame motion assumption. This modification enables the model to have negligible parameter size increment even when the number of predicted frames are high. Unlike the previous approaches, our model predicts frames in a single step without middle frame dependency. It is advantageous not only because it is simple to use but also robust to heavy blurs where middle frame prediction often fails. Overall, the simplicity and flexibility of our method makes it a promising approach for future applications such as deblurring, temporal super resolution and flow estimation from a motion-blurred image. Here, we present additional details pertaining to the experiments that could not be included in the main text due to space constraints. As can be inferred from Table 4, our method consistently performs favorably against the competing method. The prediction performs best for middle frames and consistently decreases for non-middle frames on both our method and the competing method. The overall performance reported here is consistent with the in the main paper. Here, we show qualitative (frame-by-frame comparison) from the high speed video dataset that could not be included in the main text due to space constraints. We also tested our model on motion-blurred examples from and the restored videos are shown in Fig. 9. Here, we show qualitative from the SUN360 panorama dataset (J. Xiao & Torralba., 2012) that could not be included in the main text due to space constraints. The samples contain various blur patterns e.g.static, partial blur, and heavy blur.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We present a novel unified architecture that restores video frames from a single motion-blurred image in an end-to-end manner.

Represent the short summary for finding the corresponding scientific article (long)

We propose a novel way to incorporate conditional image information into the discriminator of GANs using feature fusion that can be used for structured prediction tasks.


Question: Represent the scientific article to find a short TLDR

Question: High performance of deep learning models typically comes at cost of considerable model size and computation time. These factors limit applicability for deployment on memory and battery constraint devices such as mobile phones or embedded systems. In this work we propose a novel pruning technique that eliminates entire filters and neurons according to their relative L1-norm as compared to the rest of the network, yielding more compression and decreased redundancy in the parameters. The ing network is non-sparse, however, much more compact and requires no special infrastructure for its deployment. We prove the viability of our method by achieving 97.4%, 47.8% and 53% compression of LeNet-5, ResNet-56 and ResNet-110 respectively, exceeding state-of-the-art compression reported on ResNet without losing any performance compared to the baseline. Our approach does not only exhibit good performance, but is also easy to implement on many architectures. While deep learning models have become the method of choice for a multitude of applications, their training requires a large number of parameters and extensive computational costs (energy, memory footprint, inference time). This limits their deployment on storage and battery constraint devices, such as mobile phones and embedded systems. To compress deep learning models without loss in accuracy, previous work proposed pruning weights by optimizing network's complexity using second order derivative information BID1 BID4. While second order derivative introduces a high computational overhead, BID7 BID9 explored low rank approximations to reduce the size of the weight tensors. Another line of work BID3 BID14, proposed to prune individual layer weights with the lowest absolute value (nonstructural sparsification of layer weights). BID2 followed the same strategy while incorporating quantization and Huffman coding to further boost compression. While the aforementioned methods considered every layer independently, BID12 proposed to prune the network weights in a class-blind manner, e.g. individual layer weights are pruned according to their magnitude as compared to all weights in the network. Noteworthy, all approaches that prune weights non-structurally, generally in high sparsity models that require dedicated hardware and software. Structured pruning alleviates this by removing whole filters or neurons, producing a non-sparse compressed model. In this regard, BID11 proposed channel-wise pruning according to the L1-norm of the corresponding filter. BID15 learned a compact model based on learning structured sparsity of different parameters. A data-free algorithm was implemented to remove redundant neurons iteratively on fully connected layers in BID13. In BID6, connections leading to weak activations were pruned. Finally, BID16 pruned neurons by measuring their importance with respect to the penultimate layer. Generally, in structured pruning, each layer is pruned separately, which requires calculation of layer importance before training. This work features two key components: a) Blindness: all layers are considered simultaneously; blind pruning was first introduced by BID12 to prune individual weights; b) Structured Pruning: removal of entire filters instead of individual weights. To the best of our knowledge, we are the first to use these two components together to prune filters based on their relative L1-norm compared to the sum of all filters' L1-norms across the network, instead of pruning filters according to their L1-norm within the layer BID11, inducing a global importance score for each filter. The contribution of this paper is two-fold: i) Proposing a structured class-blind pruning technique to compress the network by removing whole filters and neurons, which in a compact non-sparse network with the same baseline performance. ii) Introducing a visualization of global filter importance to devise the pruning percentage of each layer. As a , the proposed approach achieves higher compression gains with higher accuracy compared to the state-of-the-art reported on ResNet-56 and ResNet-110 on the CIFAR10 dataset BID8. Consider a network with a convolutional (conv) layer and a fully connected (fc) layer. We denote each filter F ilter i, where i ∈ [1, F], and F is the total number of filters in the conv layer. Each filter is a 3D kernel space consisting of channels, where each channel contains 2D kernel weights. For the fc layer, we denote W m, a 1-D feature space containing all the weights connected to certain neuron N euron m, with m ∈ [1, N] and N denoting the number of neurons. It should be noted that We do not prune the classification layer. Each pruning iteration in our algorithm is structured as follows:Algorithm 1 Pruning procedure 1: for i ← 1 to F do loop over filters of a conv layer 2: DISPLAYFORM0 calculate L1-norm of all channels' kernel weights 3: if norm_conv (i) < threshold then 12: prune(F ilter i) remove filter if its normalized norm is less than threshold 13: for m ← 1 to N do 14: if norm_f c(m) < threshold then 15: prune(N euron m) remove neuron if its normalized norm is less than threshold Importance calculation. Although pre-calculation of filters or layers' sensitivity to be pruned is not needed in our method, it can be visualized as part of the pruning criteria. In our algorithm, blindness implies constructing a hidden importance score, which corresponds to the relative normalized L1-norm. For instance, the relevant importance for a certain filter in a conv layer w.r.t. all other filters in all layers is the ratio between the filter's normalized norm and the sum of all filters' normalized norms across the network. DISPLAYFORM1 Normalization. As each layer's filters have different number of kernel weights, we normalize filters' L1-norms by dividing each over the number of kernel weights corresponding to the filter (Line 3 and 6 as indicated in Algorithm 1). Alternatively without normalization, filters with a higher number of kernel weights would have higher probabilities of higher L1-norms, hence lower probability to get pruned. Retraining process. Pruning without further adaption, in performance loss. Therefore, in order to regain base performance, it is necessary for the model to be retrained. To this end, we apply an iterative pruning schedule that alternates between pruning and retraining. This is conducted until a maximum compression is reached without losing the base accuracy. BID3 0.77 92.00 84.00 Srinivas et al. BID14 0.81 95.84 91.68 Han et al. BID2 0.74 97.45 -. Table 2: Results on LeNet-5. Error% percentage for different percentage of parameters pruned (Par.%); "E.Par%" is the effective pruning percentage after adding the extra indices' storage for non-structured pruning as studied by BID0 3 ExperimentIn order to assess the efficacy of the proposed method, we evaluate the performance of our technique on a set of different networks: first, LeNet-5 on MNIST BID10; second, ResNet-56 and ResNet-110 (BID5) on CIFAR-10 BID8. We use identical training settings as BID5, after pruning we retrain with learning rate of 0.05.For ResNet, when a filter is pruned, the corresponding batch-normalization weight and bias applied on that filter are pruned accordingly. After all pruning iterations are finished, a new model with the remaining number of parameters is created. We report compression on the existing benchmark BID11 BID16. As shown in Table 1, we outperform the state-of-the-art compression reported by BID16 on both ResNet-56 and ResNet-110 with a lower classification error even compared to the baseline. In Table 2, while using one-shot pruning, the influence of our method's different components; structured pruning and blindness, is analyzed by removing a component each test, ing in: i) Non-Structured -pruning applied on weights separately. ii) Non-Blind -every layer is pruned individually. Then, the effect of the pruning strategy on the method with all its components is analyzed by comparing: i) Ours-Oneshot -using one-shot pruning and ii) Ours -using iterative pruning. By comparing the previous versions that are using one-shot pruning, our method has less number of parameters compared to the other versions; ("Non-Structured" and "Non-Blind").Finally, applying pruning iteratively is superior to one-shot pruning. We also show that our method performs better than previously mentioned non-structured weight pruning techniques BID3 BID14. Proposed structured class-blind pruning offers comparable performance as BID2, without requiring dedicated hardware and software to realize compression. We presented a novel structured pruning method to compress neural networks without losing accuracy. By pruning layers simultaneously instead of looking at each layer individually, our method combines all filters and output features of all layers and prunes them according to a global threshold. We have surpassed state-of-the-art compression reported on ResNet-56 and ResNet-110 on CIFAR-10 BID16, compressing more than 47% and 53% respectively. Also, we showed that only 11K parameters are sufficient to exceed the baseline performance on LeNet-5, compressing more than 97%. To realize the advantages of our method, no customized hardware or libraries are needed. It is worth to say that due to removing whole filters and neurons, the pruning percentage reflects the effective model compression percentage. For the future work, we are dedicated to proving the applicability of our method on several different architectures and datasets. Hence, we plan to experiment on VGG-16, ResNet on ImageNet and/or other comparable architectures.

We propose a novel structured class-blind pruning technique to produce highly compressed neural networks.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.