 in Appendix \[app:CR\].

Policy Gradient
---------------

Consider the problem of finding a good policy over a parameterized class, $\{\pi_{\btheta}: \btheta \in \Theta\}$. Each policy $\pi_{\btheta}: \Scal \to \Delta(\Acal)$ is stochastic and we assume that $\pi_\btheta(a|s)$ is differentiable w.r.t. $\btheta$. Policy gradient algorithms [@williams1992simple] perform (stochastic) gradient descent on the objective $J(\pi_\btheta)$, and the following expression is an unbiased gradient based on a single trajectory [@sutton2000policy]: $$\begin{aligned}
\label{eq:pg}
\sum_{t=0}^T \left(\gbt \log\pi_{\btheta}(a_t|s_t) \sum_{t'=t}^T \gamma^{t'} r_{t'}\right). \end{aligned}$$ Note that although most PG results are derived for the infinite-horizon discounted case, they can be immediately applied to our setup, since our formulation in Section \[sec:mdp\] can be turned into an infinite-horizon discounted MDP by treating $s_{T+1}$ as an absorbing state.

Further Notations {#sec:notation}
-----------------

Since we always consider the estimators based on a single on-policy trajectory, all expectations $\EE[\cdot]$ are w.r.t. that on-policy distribution induced by $\pi$ (for OPE) or $\pi_\btheta$ (for PG). Following the notations in @jiang2016doubly, we use $\EE_t[\cdot]$ as a shorthand for the conditional expectation $\EE[\cdot | s_0, a_0, \ldots, s_{t-1}, a_{t-1}]$, and similarly $\VV_t[\cdot]$ and $\cov_t[\cdot]$ for the conditional (co)variance. We will often see the usage $\EE_t[\cdot | s_t]$, which simply means $\EE[\cdot | s_0, a_0, \ldots, s_{t-1}, a_{t-1}, s_t]$.

Since we will talk about multiple OPE estimators, instead of giving each estimator a separate variable name, we will just use $\hv^{(\cdot)}$ as a placeholder for the O
in Appendix   \[app: CR\ ]. 

 Policy Gradient 
 --------------- 

 Consider the problem of finding a good policy over a parameterized course, $ \{\pi_{\btheta }: \btheta \in \Theta\}$. Each policy $ \pi_{\btheta }: \Scal \to \Delta(\Acal)$ is stochastic and we wear that $ \pi_\btheta(a|s)$ is differentiable w.r.t.   $ \btheta$. Policy gradient algorithms [ @williams1992simple ] perform (stochastic) gradient descent on the objective $ J(\pi_\btheta)$, and the follow formulation is an unbiased gradient based on a individual trajectory [ @sutton2000policy ]: $ $ \begin{aligned } 
 \label{eq: pg } 
 \sum_{t=0}^T \left(\gbt \log\pi_{\btheta}(a_t|s_t) \sum_{t'=t}^T \gamma^{t' } r_{t'}\right). \end{aligned}$$ Note that although most PG results are deduce for the infinite - horizon discounted case, they can be immediately apply to our setup, since our formulation in Section   \[sec: mdp\ ] can be turned into an infinite - horizon discounted MDP by treating $ s_{T+1}$ as an absorbing state of matter. 

 Further Notations { # sec: notation } 
 ----------------- 

 Since we always consider the calculator based on a single on - policy trajectory, all expectation $ \EE[\cdot]$ are w.r.t.   that on - policy distribution induced by $ \pi$ (for OPE) or $ \pi_\btheta$ (for PG). come the notations in @jiang2016doubly, we use $ \EE_t[\cdot]$ as a shorthand for the conditional expectation $ \EE[\cdot | s_0, a_0, \ldots, s_{t-1 }, a_{t-1}]$, and similarly $ \VV_t[\cdot]$ and $ \cov_t[\cdot]$ for the conditional (co)variance. We will often see the use $ \EE_t[\cdot | s_t]$, which simply means $ \EE[\cdot | s_0, a_0, \ldots, s_{t-1 }, a_{t-1 }, s_t]$. 

 Since we will talk about multiple OPE estimators, instead of giving each estimator a separate variable name, we will just use $ \hv^{(\cdot)}$ as a placeholder for the O
 in Appendix \[app:CR\].

Policy Graaient
---------------

Consider tkw probnem of finding a good policy over a parameverized coass, $\{\pi_{\btheta}: \btheta \iv \Theta\}$. Ewch polixy $\pm_{\btheta}: \Scal \to \Vslta(\Acal)$ is sflchavvic and we assuke that $\pi_\ttheta(a|s)$ is dixfdrzntiable w.r.t. $\btheta$. Policy gradient ajgorithks [@williams1992simplg] pergjrm (anoghastic) gradient descent on the kbjectine $J(\pi_\btheta)$, and yhe following expression id an unbiased gradient based on a sindoe trajectorh [@sutton2000policy]: $$\begin{aljgned}
\label{eq:pg}
\sum_{t=0}^T \left(\gbt \log\oi_{\bthzta}(a_t|s_t) \sum_{j'=c}^R \gwkma^{t'} r_{t'}\righv). \end{ajigned}$$ Note bnat aldhough kost PG resultx ace dwrived for the infinive-horizon discounted case, thef ean be immediately applued tm ous segyp, rinde ohr forluletion in Sedtion \[sec:mdp\] can be turned into ag infinite-horiaon difcjunted MDP by treating $s_{T+1}$ as an absorbitg atate.

Further Notations {#wec:notation}
-----------------

Since we apways confider the estimators based on a single on-policy tsajecvofy, col exowchations $\EE[\cdot]$ are w.r.t. that on-policy distributykn ikduced by $\pi$ (for OPE) or $\pi_\byhfts$ (for PG). Folloding tks hotations in @jiang2016foubly, re usw $\EE_t[\cdot]$ as s shorthand for the conditiinal expectanion $\EE[\cdot | s_0, a_0, \ldots, s_{t-1}, a_{t-1}]$, and rimikarly $\VV_t[\cdot]$ and $\cov_t[\cdot]$ fur tge conditiojal (co)varjxnce. We will oftdn xea the usage $\EE_t[\cdot | s_t]$, whych simplb meaus $\EE[\cdog | s_0, a_0, \ldoes, s_{t-1}, a_{t-1}, s_h]$.

Sincc we will talk abouh muljiple MPE estimahors, instead of giving each estmkator a sepatata vdriable uame, wc will just use $\hv^{(\cdot)}$ as a pkaceholber fof the O
in Appendix \[app:CR\]. Policy Gradient --------------- Consider of a good over a parameterized Each $\pi_{\btheta}: \Scal \to is stochastic and assume that $\pi_\btheta(a|s)$ is differentiable w.r.t. Policy gradient algorithms [@williams1992simple] perform (stochastic) gradient descent on the objective $J(\pi_\btheta)$, and following expression is an unbiased gradient based on a single trajectory [@sutton2000policy]: $$\begin{aligned} \sum_{t=0}^T \log\pi_{\btheta}(a_t|s_t) \gamma^{t'} \end{aligned}$$ Note that although most PG results are derived for the infinite-horizon discounted case, they can immediately applied to our setup, since our formulation Section \[sec:mdp\] can be into an infinite-horizon discounted MDP treating as an state. Notations ----------------- Since we consider the estimators based on a single on-policy trajectory, all expectations $\EE[\cdot]$ are w.r.t. that on-policy distribution by $\pi$ or $\pi_\btheta$ PG). the in @jiang2016doubly, we as a shorthand for the conditional s_0, a_0, \ldots, s_{t-1}, a_{t-1}]$, and similarly $\VV_t[\cdot]$ $\cov_t[\cdot]$ for conditional (co)variance. We will often see usage $\EE_t[\cdot | s_t]$, which simply means $\EE[\cdot s_0, a_0, \ldots, s_{t-1}, a_{t-1}, s_t]$. Since we will talk about multiple OPE estimators, instead each estimator a separate name, we will use as placeholder the O
 in Appendix \[app:CR\].

Policy GradIent
---------------

ConsidEr the ProBleM oF finDing A good policy oveR A parAmeterized class, $\{\pi_{\bthetA}: \btheTa \IN \TheTA\}$. EAch poLicy $\pi_{\bTHeTA}: \scaL \tO \DEltA(\ACAl)$ Is stoChaStic and We assume thAt $\pI_\bTheta(a|s)$ is difFErEntiable w.r.T. $\btHeta$. Policy grAdiEnt algOrIthMS [@willIamS1992simpLe] perfORm (stocHastic) graDiENt descENt on the OBJeCtivE $J(\pi_\btheta)$, and the fOLlOWing expression Is an unBiASeD GRadIenT based on a sInGle trAJectory [@SUtTON2000PolICy]: $$\begin{aligneD}
\label{eq:pg}
\sUM_{t=0}^T \Left(\gbT \lOg\pI_{\Btheta}(A_t|s_t) \sUm_{T'=T}^T \gAmma^{t'} r_{t'}\righT). \end{Aligned}$$ NoTe that ALthough MOst PG reSults aRe dEriVed fOR tHe InfInITe-hORiZon DIscOunted caSe, ThEy can Be imMEDIAtelY apPlieD to ouR setup, since ouR foRmulATioN in SeCtion \[Sec:mDp\] Can be Turned Into aN iNfinite-horizon dIscoUnted MDP bY trEaTinG $s_{t+1}$ as an ABsorbiNg sTatE.

FurtheR NotatiONs {#sEc:NOTAtIon}
-----------------

Since we always coNsIDEr The estimAtors bASeD oN A single oN-pOliCy trAJEctorY, all EXpEctationS $\EE[\cdoT]$ ArE w.R.t. that oN-pOlicy dIsTriButIon inDUced By $\pi$ (foR OPE) or $\pi_\BthetA$ (For PG). Following THe notations in @JIaNG2016DoUBly, wE usE $\EE_t[\cdot]$ as a ShorTHand For tHE cOndITionaL expeCtATiON $\EE[\cdot | s_0, a_0, \ldots, s_{t-1}, a_{t-1}]$, aNd SimilaRly $\VV_T[\cdot]$ and $\cov_t[\cDot]$ for the cONDItional (cO)varIAnCE. We will often seE the uSage $\EE_t[\cdoT | S_t]$, which sImply Means $\EE[\cDot | s_0, a_0, \ldotS, S_{T-1}, a_{t-1}, s_t]$.

SinCe wE wiLl tAlk ABOuT multiple OPE eSTImatOrS, insteaD of Giving eAch EstImaTor A sEparate vaRiable naMe, We WiLl JusT use $\hV^{(\Cdot)}$ as a pLaCehOlDer For thE o
 in Appendix \[app:CR\].
Policy Gra dient 
-- --- -- ---- ---- 

Consider the prob lem of finding a goodpolic yo vera  p arame terized cl a s s,$\ {\ pi_ {\ b th eta}:  \b theta \ in \Theta\ }$.  E ach policy $ \ pi _{\btheta} : \ Scal \to \De lta (\Acal )$  is stoch ast ic an d we a s sume t hat $\pi_ \b t heta(a | s)$ isd i ff eren tiable w.r.t. $\b t he t a$. Policy gra diental g or i t hms  [@ williams19 92 simpl e ] perfo r m( s t och a stic) gradien t descent o n  th e obje ct ive $J(\pi _\bth et a )$,  and the fo llow ing expre ssioni s an un b iased g radien t b ase d on asi ngl et raj e ct ory [@s utton200 0p ol icy]:  $$\ b e g i n{al ign ed}\labe l{eq:pg}
\sum _{t =0}^ T  \l eft(\ gbt \ log\ pi _{\bt heta}( a_t|s _t ) \sum_{t'=t}^T  \ga mma^{t'}r_{ t' }\r ig ht).\ end{al ign ed} $$ Note  that a l tho ug h m os t PG results are d er i v ed  for the  infin i te -h o rizon di sc oun tedc a se, t heyc an  be imme diatel y  a pp lied to  o ur set up , s inc e our form ulatio n in Sec tion\ [sec:mdp\] can be turned int o  a n in f init e-h orizon disc ount e d MD P by tr eat i ng $s _{T+1 }$ as an absorbing state. 
Furthe r Not ations {#sec: notation}- - - -------- ---- - -Since we alway s con sider thee stimator s bas ed on asingle on - p olicy tr aje cto ry,  al l ex pectations $\ E E [\cd ot ]$ arew.r .t. tha t o n-p oli cydi stributio n induce dby  $ \p i$(forO PE) or $ \p i_\ bt het a$ (f o r PG).  Foll owin gth e  no tations in @ jian g2 01 6dou bly ,we us e $\ E E_t [\cdot] $ as a sh ort h andfo rthe con ditional expe ct ation $\EE [\ cdo t | s_ 0 ,  a_0, \l dots, s_{t-1}, a_{t-1}] $ , and s imi larly  $\V V_t[\cdot ]$and $\ cov _ t[\cdo t]$ fo r the  c ond i t ional ( co )va ri ance. We w i l l o ftense e th e usage  $\EE_t[\cdot | s_ t ]$,  which simply  me ans$ \ EE [\c d ot | s _0 ,  a_ 0 ,  \ldots, s_{t-1 }, a_{t-1} ,s _t ]$.

Since wewi ll talk  aboutmulti p le OPEestimator s, instea dof g i v ing  each esti mator aseparatev ariab l ename,  we  willju stuse $ \hv^{( \ cdo t)}$as a p la cehold er fo rthe O
 in_Appendix \[app:CR\].

Policy Gradient
---------------

Consider_the problem of finding_a good_policy_over a_parameterized_class, $\{\pi_{\btheta}: \btheta_\in \Theta\}$. Each_policy $\pi_{\btheta}: \Scal \to_\Delta(\Acal)$ is stochastic_and_we assume that $\pi_\btheta(a|s)$ is differentiable w.r.t. $\btheta$. Policy gradient algorithms [@williams1992simple] perform (stochastic) gradient_descent_on the_objective_$J(\pi_\btheta)$,_and the following expression is_an unbiased gradient based on_a single_trajectory [@sutton2000policy]: $$\begin{aligned}
\label{eq:pg}
\sum_{t=0}^T \left(\gbt \log\pi_{\btheta}(a_t|s_t) \sum_{t'=t}^T \gamma^{t'} r_{t'}\right)._\end{aligned}$$_Note that although_most PG results are derived for the infinite-horizon discounted_case, they can be immediately applied_to our setup,_since_our_formulation in Section \[sec:mdp\] can_be turned into an infinite-horizon discounted_MDP by treating $s_{T+1}$ as an_absorbing state.

Further Notations {#sec:notation}
-----------------

Since we always consider_the estimators based on a single_on-policy trajectory, all expectations $\EE[\cdot]$_are w.r.t. that_on-policy distribution induced by $\pi$_(for OPE) or_$\pi_\btheta$ (for_PG). Following the_notations in @jiang2016doubly, we use $\EE_t[\cdot]$_as a shorthand_for the conditional expectation $\EE[\cdot |_s_0,_a_0, \ldots, s_{t-1},_a_{t-1}]$,_and_similarly $\VV_t[\cdot]$_and $\cov_t[\cdot]$ for_the_conditional (co)variance._We_will often see the usage $\EE_t[\cdot_|_s_t]$, which simply means $\EE[\cdot | s_0,_a_0, \ldots, s_{t-1}, a_{t-1},_s_t]$.

Since_we will talk about_multiple OPE estimators, instead of_giving each estimator a separate variable_name, we_will just_use $\hv^{(\cdot)}$ as a placeholder for the O