 “data” at $x_0$ (i.e., derivatives $f^{(i)}(x_0)$).

Our paper proceeds as follows. In Section \[sec:terl\], we start with a general result of applying Taylor expansions to Q-functions. When we apply the same technique to the RL objective, we reuse the general result and derive a higher-order policy optimization objective. This leads to Section \[sec:TayPO\], where we formally present the *Taylor Expansion Policy Optimization* (TayPO) and generalize prior work [@schulman2015trust; @schulman2017proximal] as a first-order special case. In Section \[sec:uni\], we make clear connection between Taylor expansions and $Q(\lambda)$ [@harutyunyan_QLambda_2016], a common return-based off-policy evaluation operator. Finally, in Section \[sec:exp\], we show the performance gains due to the higher-order objectives across a range of state-of-the-art distributed deep RL agents.

Taylor expansion for reinforcement learning {#sec:terl}
===========================================

Consider a Markov Decision Process (MDP) with state space $\mathcal{X}$ and action space $\mathcal{A}$. Let policy $\pi(\cdot|x)$ be a distribution over actions give state $x$. At a discrete time $t \geq 0$, the agent in state $x_t$ takes action $a_t \sim \pi(\cdot|x_t)$, receives reward $r_t \triangleq  r(x_t,a_t)$, and transitions to a next state $x_{t+1} \sim p(\cdot|x_t,a_t)$. We assume a discount factor $\gamma \in [0,1)$. Let $Q^\pi(x,a)$ be the action value function (Q-function) from state $x,$ taking action $a,$ and following policy $\pi$. For convenience, we use $ d_\gamma^\pi(\cdot,\cdot|x_0,a_0,\tau)$ to denote the discounted visitation distribution starting from state-action pair $(x_0,a_0)$ and following $\pi$, such that $d_\gamma^\pi(x,a|x_0,a_0,\tau) = (1-\gamma)\gamma^{-\tau} \sum_{t\geq \tau} \
“ data ” at $ x_0 $ (i.e., derivatives $ f^{(i)}(x_0)$). 

 Our paper proceeds as follows. In Section   \[sec: terl\ ], we depart with a cosmopolitan result of applying Taylor expansion to Q - function. When we apply the same proficiency to the RL objective, we reuse the general consequence and derive a higher - club policy optimization objective. This leads to Section   \[sec: TayPO\ ], where we formally award the * Taylor Expansion Policy Optimization * (TayPO) and generalize prior work [ @schulman2015trust; @schulman2017proximal ] as a first - order special event. In Section   \[sec: uni\ ], we make clear connection between Taylor expansion and $ Q(\lambda)$ [ @harutyunyan_QLambda_2016 ], a common return - based off - policy evaluation operator. ultimately, in Section   \[sec: exp\ ], we show the performance gains due to the higher - order aim across a range of state - of - the - art distributed deep RL agents. 

 Taylor expansion for reinforcement learning { # sec: terl } 
 = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 

 Consider a Markov Decision Process (MDP) with state space $ \mathcal{X}$ and action space $ \mathcal{A}$. Let policy $ \pi(\cdot|x)$ be a distribution over actions give state $ x$. At a discrete clock time $ t \geq 0 $, the agent in department of state $ x_t$ takes action $ a_t \sim \pi(\cdot|x_t)$, receives reward $ r_t \triangleq   r(x_t, a_t)$, and conversion to a future state $ x_{t+1 } \sim p(\cdot|x_t, a_t)$. We assume a deduction factor $ \gamma \in [ 0,1)$. Let $ Q^\pi(x, a)$ be the action value function (Q - function) from state $ x,$ necessitate action $ a,$ and following policy $ \pi$. For convenience, we use $ d_\gamma^\pi(\cdot,\cdot|x_0,a_0,\tau)$ to announce the discounted trial distribution starting from state - action pair $ (x_0,a_0)$ and following $ \pi$, such that $ d_\gamma^\pi(x, a|x_0,a_0,\tau) = (1-\gamma)\gamma^{-\tau } \sum_{t\geq \tau } \
 “daha” at $x_0$ (i.e., derivatives $f^{(l)}(x_0)$).

Our paper procgees as hollows. In Sectkon \[sec:terl\], we start with a gxnerql rewult of applying Taylof expansilns to Q-duncuions. When we apply the same tedmniquz vo the RL objecjive, we reusa the general serupt and derive a higher-order policy jptimizstlon objective. Jhis kqads no Section \[sec:TayPO\], where we forjally pgesent the *Taylor Expansion Policy Optimizahion* (TayPO) and generallze prior wirk [@fxhulman2015trust; @schulman2017pgmximal] as z first-order special case. In Seztion \[xec:uni\], we naje fnear connecvion bvtween Taylor expansimns and $Q(\lambda)$ [@harutnunyai_QLanbda_2016], a common return-besed off-policy evaluwtion opesacor. Finally, in Sectiob \[swc:exp\], we vhow rhe pedfprjance haiis due to tge higher-oreer objectives acroxs q range of stzte-of-tre-wrt distributed deep RL agents.

Taylor exkansikn for reinforcement leqrning {#sec:terl}
===========================================

Considet a Markov Decision Process (MDP) with state space $\mathcal{X}$ atd acviun wpwzw $\lathcal{A}$. Let policy $\pi(\cdot|x)$ be a distribution kvtr sctions give sbate $x$. At a discreye toie $t \geq 0$, the agent in state $x_t$ takes achion $a_t \sim \pi(\cdot|x_t)$, weceoves reward $r_t \triangleq  r(z_t,a_t)$, and trausirions to a next stcte $x_{t+1} \sim p(\edot|x_t,s_t)$. We assume a discount factur $\gzmma \in [0,1)$. Leh $Q^\pi(x,a)$ bs the action valud flncthon (Q-function) from state $x,$ taking artion $a,$ and wollpwing [olicy $\pi$. Vor convenience, we use $ f_\gammc^\pi(\cdmt,\cdot|x_0,a_0,\tak)$ to denote the discounted visivetion distribotimn vtarting from state-action [air $(x_0,a_0)$ and foklowing $\pi$, sjch that $d_\famma^\pi(e,a|x_0,a_0,\tau) = (1-\gamia)\gamma^{-\tau} \suk_{j\geq \tau} \
“data” at $x_0$ (i.e., derivatives $f^{(i)}(x_0)$). Our as In Section we start with Taylor to Q-functions. When apply the same to the RL objective, we reuse general result and derive a higher-order policy optimization objective. This leads to Section where we formally present the *Taylor Expansion Policy Optimization* (TayPO) and generalize prior [@schulman2015trust; as first-order case. In Section \[sec:uni\], we make clear connection between Taylor expansions and $Q(\lambda)$ [@harutyunyan_QLambda_2016], a common off-policy evaluation operator. Finally, in Section \[sec:exp\], we the performance gains due the higher-order objectives across a of distributed deep agents. expansion reinforcement learning {#sec:terl} Consider a Markov Decision Process (MDP) with state space $\mathcal{X}$ and action space $\mathcal{A}$. Let policy $\pi(\cdot|x)$ a distribution give state At discrete $t \geq 0$, in state $x_t$ takes action $a_t reward $r_t \triangleq r(x_t,a_t)$, and transitions to a state $x_{t+1} p(\cdot|x_t,a_t)$. We assume a discount factor \in [0,1)$. Let $Q^\pi(x,a)$ be the action value (Q-function) from state $x,$ taking action $a,$ and following policy $\pi$. For convenience, we use to denote the discounted distribution starting from pair and $\pi$, that $d_\gamma^\pi(x,a|x_0,a_0,\tau) (1-\gamma)\gamma^{-\tau} \sum_{t\geq \tau} \
 “data” at $x_0$ (i.e., derivatives $f^{(i)}(x_0)$).

OuR paper procEeds aS foLloWs. in SeCtioN \[sec:terl\], we starT With A general result of applyiNg TayLoR ExpaNSiOns to q-functiONs. wHEn wE aPpLy tHe SAmE techNiqUe to the rL objectivE, we ReUse the generaL ReSult and derIve A higher-order PolIcy optImIzaTIon obJecTive. THis leaDS to SecTion \[sec:TaYPo\], Where wE FormallY PReSent The *Taylor ExpansioN poLIcy OptimizatioN* (TayPO) AnD GeNERalIze Prior work [@sChUlman2015TRust; @schULmAN2017PRoxIMal] as a first-orDer special cASe. IN SectiOn \[Sec:UNi\], we maKe cleAr COnnEction betweEn TaYlor expanSions aND $Q(\lambdA)$ [@HarutyuNyan_QLAmbDa_2016], a CommON rEtUrn-BaSEd oFF-pOliCY evAluation OpErAtor. FInalLY, IN sectIon \[Sec:eXp\], we sHow the performAncE gaiNS duE to thE highEr-orDeR objeCtives AcrosS a Range of state-of-tHe-arT distribuTed DeEp Rl aGents.

tAylor eXpaNsiOn for reInforceMEnt LeARNInG {#sec:terl}
===========================================

Consider a MArKOV DEcision PRocess (mdP) WiTH state spAcE $\maThcaL{x}$ And acTion SPaCe $\mathcaL{A}$. Let pOLiCy $\Pi(\cdot|x)$ Be A distrIbUtiOn oVer acTIons Give stAte $x$. At a dIscreTE time $t \geq 0$, the agENt in state $x_t$ taKEs ACTiON $a_t \sIm \pI(\cdot|x_t)$, receIves REwarD $r_t \tRIaNglEQ  r(x_t,a_T)$, and tRaNSiTIons to a next state $x_{t+1} \sIm P(\cdot|x_T,a_t)$. We Assume a discouNt factor $\gaMMA \In [0,1)$. Let $Q^\pi(X,a)$ be THe ACtion value funcTion (Q-Function) frOM state $x,$ tAking Action $a,$ aNd followiNG Policy $\pi$. for ConVenIenCE, We Use $ d_\gamma^\pi(\cdOT,\Cdot|X_0,a_0,\Tau)$ to deNotE the disCouNteD viSitAtIon distriBution stArTiNg FrOm sTate-aCTion pair $(X_0,a_0)$ And FoLloWing $\pI$, Such thAt $d_\gaMma^\pI(x,A|x_0,A_0,\Tau) = (1-\Gamma)\gaMMa^{-\TAU} \sum_{T\gEq \Tau} \
 “data” at $x_0$ (i.e., de rivatives$f^{( i)} (x_ 0) $).
Our  paper proceed s  asfollows. In Section \[ sec:t er l \],w estart  with a ge n e ral  r es ult  o f  a pplyi ngTaylorexpansions  to  Q -functions.W he n we apply  th e same techn iqu e to t he  RL objec tiv e, we  reuse the ge neral res ul t  and d e rive ah i gh er-o rder policy optim i za t ion objective.  Thisle a ds t o S ect ion \[sec: Ta yPO\] ,  wherew ef o r mal l y present the  *Taylor Ex p ans ion Po li cyO ptimiz ation *( Tay PO) and gen eral ize prior  work[ @schulm a n2015tr ust; @ sch ulm an20 1 7p ro xim al ]  as afir s t-o rder spe ci al  case . In S e c tion  \[ sec: uni\] , we make cle arconn e cti on be tween  Tay lo r exp ansion s and  $ Q(\lambda)$ [@h arut yunyan_QL amb da _20 16 ], ac ommonret urn -basedoff-pol i cyev a l u at ion operator. Fina ll y ,  i n Sectio n \[se c :e xp \ ], we sh ow  th e pe r f orman ce g a in s due to  the h i gh er -orderob jectiv es  ac ros s a r a ngeof sta te-of-th e-art distributed de e p RL agents.Ta y l or expa nsi on for rein forc e ment  lea r ni ng{ #sec: terl} 
= = == = =================== == ====== ===== ======

Consi der a Mark o v Decision  Pro c es s  (MDP) with st ate s pace $\mat h cal{X}$and a ction sp ace $\mat h c al{A}$.Let  po lic y $ \ p i( \cdot|x)$ bea dist ri butionove r actio nsgiv e s tat e$x$. At a  discret eti me  $ t \ geq 0 $ , the ag en t i nsta te $x _ t$ tak es ac tion  $ a_ t  \s im \pi( \ cd o t |x_t )$ ,rece ive srewar d $r _ t \ triangl eq  r(x_t ,a_ t )$,an dtransit ions to a nex tstate $x_{ t+ 1}\sim p ( \ cdot|x_t ,a_t)$. We assume a dis c ount fa cto r $\g amma  \in [0,1 )$.  Let $ Q^\ p i(x,a) $ be t he ac ti onv a lue f u n ct ion  ( Q-function ) fro m sta te  $x, $ takin g action $a,$ andf oll owing policy$\p i$.F o rcon v en i enc e, weu s e $ d_\gamma^\p i(\cdot,\c do t |x _0,a_0,\ta u )$to  denote  the di scoun t ed visi tation di stributio nstar t i ngfrom state -actionpair $(x_ 0 ,a_0) $  a nd fo llo wing $ \p i$,  such  that$ d_\ gamma ^\pi(x ,a |x_0,a _0,\t au ) = (1-\ gamma)\gamma^{-\tau} \s um_{t\ geq \ tau } \
 “data”_at $x_0$_(i.e., derivatives $f^{(i)}(x_0)$).

Our paper_proceeds as_follows._In Section \[sec:terl\],_we_start with a_general result of_applying Taylor expansions to_Q-functions. When we_apply_the same technique to the RL objective, we reuse the general result and derive_a_higher-order policy_optimization_objective._This leads to Section \[sec:TayPO\], where_we formally present the *Taylor_Expansion Policy_Optimization* (TayPO) and generalize prior work [@schulman2015trust; @schulman2017proximal]_as_a first-order special_case. In Section \[sec:uni\], we make clear connection between Taylor_expansions and $Q(\lambda)$ [@harutyunyan_QLambda_2016], a common_return-based off-policy evaluation_operator._Finally,_in Section \[sec:exp\], we show_the performance gains due to the_higher-order objectives across a range of_state-of-the-art distributed deep RL agents.

Taylor expansion for_reinforcement learning {#sec:terl}
===========================================

Consider a Markov Decision_Process (MDP) with state space_$\mathcal{X}$ and_action space $\mathcal{A}$. Let policy_$\pi(\cdot|x)$ be a_distribution over_actions give state_$x$. At a discrete time $t_\geq 0$, the_agent in state $x_t$ takes action_$a_t_\sim \pi(\cdot|x_t)$, receives_reward_$r_t_\triangleq _r(x_t,a_t)$, and transitions_to_a next_state_$x_{t+1} \sim p(\cdot|x_t,a_t)$. We assume a_discount_factor $\gamma \in [0,1)$. Let $Q^\pi(x,a)$ be_the action value function_(Q-function)_from state $x,$ taking_action $a,$ and following policy_$\pi$. For convenience, we use $_d_\gamma^\pi(\cdot,\cdot|x_0,a_0,\tau)$ to_denote the_discounted visitation distribution starting from state-action pair $(x_0,a_0)$ and following $\pi$,_such that $d_\gamma^\pi(x,a|x_0,a_0,\tau) = (1-\gamma)\gamma^{-\tau} \sum_{t\geq_\tau} \