 \sum_{t=0}^T R^t(c - \tilde{g})$. The resulting algorithm, which we refer to as `ApproxPE`, is detailed in Algorithm \[alg:fast\_pe\]. Note that $\tilde{g}$ and $\tilde{h}$ can be computed in $\mathcal{O}(L d^4)$ and ${\mathcal{O}}(T d^4)$ time, respectively. Since $g$ and $h$ are equal to the limits of $\tilde{g}$ and $\tilde{h}$ as $L, T \rightarrow \infty$, we expect that larger $L$ and $T$ will yield better approximations. One must ensure that the $L$ and $T$ that are required for a good approximation do not grow too quickly with $d$. Using Proposition \[prop:properties\_of\_entropic\_tc\], we show that this is the case.

[prop]{}[policyevaluationcomplexity]{} \[prop:policy\_evaluation\_complexity\] Let $P$ and $Q$ be aperiodic and irreducible transition matrices, $R \in {\Pi_\eta(P,Q)}$ and $\varepsilon > 0$. Furthermore, let $g \in {\mathbb{R}}^{d^2}$ and $h \in {\mathbb{R}}^{d^2}$ be the gain and bias of $R$, respectively. Then for appropriate choice of $L$ and $T$, $\emph{\texttt{ApproxPE}}(R, L, T)$ yields $(\tilde{g}, \tilde{h})$ such that $\|\tilde{g} - g\|_\infty \leq \varepsilon$ and $\|\tilde{h} - h\|_1 \leq \varepsilon$ in $\tilde{\mathcal{O}}(d^4 \log \varepsilon^{-1})$ time.

In particular, `ApproxPE` does approximate `ExactPE` in time scaling like $\tilde{{\mathcal{O}}}(d^4)$. Explicit choices of $L$ and $T$ are given in the proof of Proposition \[prop:policy\_evaluation\_complexity\], which may be found in Appendix \[app:complexity\].

Entropic Policy Improvement {#sec:entropic_regularization}
---------------------------

Next we describe a means of approximating Algorithm \[alg:exact\_pi\]. Note that since the gain vector for any element of ${\Pi_\eta(P,Q)}$ is constant, we need only improve policies with respect
\sum_{t=0}^T R^t(c - \tilde{g})$. The resulting algorithm, which we refer to as 'ApproxPE `, is detailed in Algorithm \[alg: fast\_pe\ ]. Note that $ \tilde{g}$ and $ \tilde{h}$ can be calculate in $ \mathcal{O}(L d^4)$ and $ { \mathcal{O}}(T d^4)$ meter, respectively. Since $ g$ and $ h$ are equal to the limits of $ \tilde{g}$ and $ \tilde{h}$ as $ lambert, T \rightarrow \infty$, we have a bun in the oven that larger $ L$ and $ T$ will yield good approximations. One must ensure that the $ L$ and $ T$ that are necessitate for a good approximation do not develop too quickly with $ d$. use Proposition \[prop: properties\_of\_entropic\_tc\ ], we usher that this is the case. 

 [ prop]{}[policyevaluationcomplexity ] { } \[prop: policy\_evaluation\_complexity\ ] Let $ P$ and $ Q$ be aperiodic and irreducible transition matrices, $ R \in { \Pi_\eta(P, Q)}$ and $ \varepsilon > 0$. Furthermore, permit $ g \in { \mathbb{R}}^{d^2}$ and $ h \in { \mathbb{R}}^{d^2}$ be the gain and diagonal of $ R$, respectively. Then for appropriate choice of $ L$ and $ T$, $ \emph{\texttt{ApproxPE}}(R, L, T)$ yields $ (\tilde{g }, \tilde{h})$ such that $ \|\tilde{g } - g\|_\infty \leq \varepsilon$ and $ \|\tilde{h } - h\|_1 \leq \varepsilon$ in $ \tilde{\mathcal{O}}(d^4 \log \varepsilon^{-1})$ clock time. 

 In particular, 'ApproxPE 'does approximate 'ExactPE 'in clock time scaling like $ \tilde{{\mathcal{O}}}(d^4)$. Explicit choices of $ L$ and $ T$ are given in the proof of Proposition \[prop: policy\_evaluation\_complexity\ ], which may be find in Appendix \[app: complexity\ ]. 

 Entropic Policy Improvement { # sec: entropic_regularization } 
 --------------------------- 

 Next we describe a means of approximating Algorithm \[alg: exact\_pi\ ]. Note that since the gain vector for any chemical element of $ { \Pi_\eta(P, Q)}$ is constant, we need only improve policies with respect
 \sul_{t=0}^T R^t(c - \tilde{g})$. The resuuting algorithm, which xe refed to as `XpproxPE`, is detailed in Algocithn \[alg:dast\_pe\]. Note that $\tilde{e}$ and $\tilfe{h}$ can ve cinputed in $\mathcal{O}(L d^4)$ ans ${\matkcel{O}}(T d^4)$ time, reskectively. Sitce $g$ and $h$ ara dqbal to the limits of $\tilde{g}$ and $\tildq{h}$ as $L, T \rightarrow \insty$, eq exlvcu that larger $L$ and $T$ will yield getter epproximations. Pne must ensure that the $L$ and $T$ that are requirfd for a goid a[proximation ao not grow too quickli with $d$. Using Proposition \[prop:pfopercies\_of\_entroknx\_tc\], fe show thav this is the case.

[igop]{}[poliwyevalustioncomplexitn]{} \[pro':polucy\_evaluation\_complexivy\] Let $P$ and $Q$ be apgriodic ang nrreducible transitiob natriwes, $S \in {\Pi_\ega(P,S)}$ end $\varepdilkn > 0$. Furthsrmore, let $t \in {\mathbb{R}}^{d^2}$ and $h \ig {\mathbb{R}}^{d^2}$ be fhe gayn and bias of $R$, respectively. Then for apkroprjate choice of $L$ and $T$, $\wmph{\texttt{ApproxPE}}(R, L, T)$ yields $(\tilde{g}, \tilde{h})$ such that $\|\tilde{g} - g\|_\infty \leq \varepvilon$ xnd $\|\tllde{f} - j\|_1 \leq \varepsilon$ in $\tilde{\mathcal{O}}(d^4 \log \varepsymom^{-1})$ nime.

In particular, `ApproxPE` dped skproximate `ExaztPE` iu tjme scaling like $\tllde{{\matrcal{O}}}(e^4)$. Expliciu chooces of $L$ and $T$ are given ib the proof jd Proposition \[prop:'olicy\_evaluacion\_cokplexoty\], which may be found nn Appsndix \[app:colplexity\].

Ehgropic Policy Imoronemett {#sec:enufopic_regularizatijn}
---------------------------

Next we deseribe a oeanx of a[proximatijg Alnmrithm \[alg:exact\_pi\]. Jote jhat shnce the gwin vector for any element of ${\Pm_\xta(P,Q)}$ is consjand, wv need onjy imirove policies rith respect
\sum_{t=0}^T R^t(c - \tilde{g})$. The resulting algorithm, refer as `ApproxPE`, detailed in Algorithm $\tilde{h}$ be computed in d^4)$ and ${\mathcal{O}}(T time, respectively. Since $g$ and $h$ equal to the limits of $\tilde{g}$ and $\tilde{h}$ as $L, T \rightarrow \infty$, expect that larger $L$ and $T$ will yield better approximations. One must ensure the and that required for a good approximation do not grow too quickly with $d$. Using Proposition \[prop:properties\_of\_entropic\_tc\], we that this is the case. [prop]{}[policyevaluationcomplexity]{} \[prop:policy\_evaluation\_complexity\] Let and $Q$ be aperiodic irreducible transition matrices, $R \in and > 0$. let \in and $h \in be the gain and bias of $R$, respectively. Then for appropriate choice of $L$ and $T$, $\emph{\texttt{ApproxPE}}(R, T)$ yields such that - \leq and $\|\tilde{h} - \varepsilon$ in $\tilde{\mathcal{O}}(d^4 \log \varepsilon^{-1})$ time. does approximate `ExactPE` in time scaling like $\tilde{{\mathcal{O}}}(d^4)$. choices of and $T$ are given in the of Proposition \[prop:policy\_evaluation\_complexity\], which may be found in \[app:complexity\]. Entropic Policy Improvement {#sec:entropic_regularization} --------------------------- Next we describe a means of approximating Algorithm \[alg:exact\_pi\]. since the gain vector any element of is we only policies with
 \sum_{t=0}^T R^t(c - \tilde{g})$. The resultinG algorithm, Which We rEfeR tO as `APproXPE`, is detailed iN algoRithm \[alg:fast\_pe\]. Note that $\Tilde{G}$ aND $\tilDE{h}$ Can be ComputeD In $\MAThcAl{o}(L D^4)$ anD ${\mAThCal{O}}(T D^4)$ tiMe, respeCtively. SinCe $g$ AnD $h$ are equal to THe Limits of $\tiLde{G}$ and $\tilde{h}$ as $l, T \rIghtarRoW \inFTy$, we eXpeCt thaT largeR $l$ and $T$ wIll yield bEtTEr apprOXimatioNS. onE musT ensure that the $L$ anD $t$ tHAt are required fOr a gooD aPPrOXImaTioN do not grow ToO quicKLy with $d$. uSiNG pRopOSition \[prop:proPerties\_of\_enTRopIc\_tc\], we ShOw tHAt this Is the CaSE.

[prOp]{}[policyevaLuatIoncompleXity]{} \[prOP:policy\_EValuatiOn\_compLexIty\] let $P$ ANd $q$ bE apErIOdiC AnD irREduCible traNsItIon maTricES, $r \IN {\Pi_\eTa(P,q)}$ and $\VarepSilon > 0$. FurthermOre, Let $g \IN {\maThbb{R}}^{D^2}$ and $h \In {\maThBb{R}}^{d^2}$ bE the gaIn and BiAs of $R$, respectiveLy. ThEn for apprOprIaTe cHoIce of $l$ And $T$, $\emPh{\tExtTt{ApproXPE}}(R, L, T)$ yIEldS $(\tILDE{g}, \Tilde{h})$ such that $\|\tildE{g} - G\|_\INfTy \leq \varEpsiloN$ AnD $\|\tILde{h} - h\|_1 \leq \VaRepSiloN$ IN $\tildE{\matHCaL{O}}(d^4 \log \vaRepsilON^{-1})$ tImE.

In partIcUlar, `ApPrOxPe` doEs appROximAte `ExaCtPE` in tiMe scaLIng like $\tilde{{\maTHcal{O}}}(d^4)$. ExpliciT ChOICeS Of $L$ aNd $T$ Are given in tHe prOOf of propOSiTioN \[Prop:pOlicy\_EvALuATion\_complexity\], which MaY be fouNd in APpendix \[app:comPlexity\].

EntROPIc Policy imprOVeMEnt {#sec:entropic_RegulArization}
---------------------------

NEXt we descRibe a Means of aPproximatING AlgoritHm \[aLg:eXacT\_pi\]. nOTe That since the gAIN vecToR for any EleMent of ${\PI_\etA(P,Q)}$ Is cOnsTaNt, we need oNly improVe PoLiCiEs wIth reSPect
 \sum_{t=0}^T R^t(c - \til de{g})$. T he re sul tin galgo rith m, which we re f er t o as `ApproxPE`, is de taile di n Al g or ithm\[alg:f a st \ _ pe\ ].  N ote  t h at  $\ti lde {g}$ an d $\tilde{ h}$  c an be comput e din $\mathc al{ O}(L d^4)$ a nd${\mat hc al{ O }}(Td^4 )$ ti me, re s pectiv ely. Sinc e$ g$ and $h$ are e qu al t o the limits of $ \ ti l de{g}$ and $\t ilde{h }$ as $ L,T \ rightarrow  \ infty $ , we ex p ec t t hat larger $L$ an d $T$ willy iel d bett er  ap p roxima tions .O nemust ensure  tha t the $L$  and $ T $ thata re requ ired f ora g ooda pp ro xim at i ond onot gro w too qu ic kl y wit h $d $ . U sing  Pr opos ition  \[prop:prope rti es\_ o f\_ entro pic\_ tc\] ,we sh ow tha t thi sis the case.

[ prop ]{}[polic yev al uat io ncomp l exity] {}\[p rop:pol icy\_ev a lua ti o n \ _c omplexity\] Let $P $a n d$Q$ be a period i can d  irreduc ib letran s i tionmatr i ce s, $R \i n {\Pi _ \e ta (P,Q)}$  a nd $\v ar eps ilo n > 0 $ . Fu rtherm ore, let  $g \ i n {\mathbb{R}} ^ {d^2}$ and $h \i n {\ m athb b{R }}^{d^2}$ b e th e  gai n an d  b ias of $R $, re sp e ct i vely. Then for appr op riatechoic e of $L$ and$T$, $\emp h { \ texttt{A ppro x PE } }(R, L, T)$ yi elds$(\tilde{g } , \tilde {h})$  such th at $\|\ti l d e{g} - g \|_ \in fty  \l e q  \ varepsilon$ a n d  $\| \t ilde{h}  -h\|_1 \ leq  \v are psi lo n$ in $\t ilde{\ma th ca l{ O} }(d ^4 \l o g \varep si lon ^{ -1} )$ ti m e.

In  part icul ar ,` App roxPE`d oe s appr ox im ate`Ex ac tPE`in t i mescaling  like $\t ild e {{\m at hc al{O}}} (d^4)$. Expli ci t choicesof  $L $ and$ T $ are gi ven in the proof of Pro p osition  \[ prop: poli cy\_evalu ati on\_co mpl e xity\] , whic h may  b e f o u nd in A pp end ix  \[app:com p l exi ty\]. 
Entr opic Po licy Improvement { # sec :entropic_reg ula riza t i on }
- - -- - --- -- - --- - - -----------

Ne xt we desc ri b ea means of app ro ximatin g Algor ithm\ [alg:ex act\_pi\] . Note th at  sin c e  th e gain vec tor forany eleme n t of$ {\ Pi_\e ta( P,Q)}$  i s c onsta nt, we nee d onl y impr ov e poli cieswi th respe ct
 \sum_{t=0}^T_R^t(c -_\tilde{g})$. The resulting algorithm,_which we_refer_to as_`ApproxPE`,_is detailed in_Algorithm \[alg:fast\_pe\]. Note_that $\tilde{g}$ and $\tilde{h}$_can be computed_in_$\mathcal{O}(L d^4)$ and ${\mathcal{O}}(T d^4)$ time, respectively. Since $g$ and $h$ are equal to_the_limits of_$\tilde{g}$_and_$\tilde{h}$ as $L, T \rightarrow_\infty$, we expect that larger_$L$ and_$T$ will yield better approximations. One must ensure_that_the $L$ and_$T$ that are required for a good approximation do_not grow too quickly with $d$._Using Proposition \[prop:properties\_of\_entropic\_tc\],_we_show_that this is the_case.

[prop]{}[policyevaluationcomplexity]{} \[prop:policy\_evaluation\_complexity\] Let $P$ and $Q$_be aperiodic and irreducible transition matrices,_$R \in {\Pi_\eta(P,Q)}$ and $\varepsilon > 0$._Furthermore, let $g \in {\mathbb{R}}^{d^2}$ and_$h \in {\mathbb{R}}^{d^2}$ be the_gain and_bias of $R$, respectively. Then_for appropriate choice_of $L$_and $T$, $\emph{\texttt{ApproxPE}}(R,_L, T)$ yields $(\tilde{g}, \tilde{h})$ such_that $\|\tilde{g} -_g\|_\infty \leq \varepsilon$ and $\|\tilde{h} -_h\|_1_\leq \varepsilon$ in_$\tilde{\mathcal{O}}(d^4_\log_\varepsilon^{-1})$ time.

In_particular, `ApproxPE` does_approximate_`ExactPE` in_time_scaling like $\tilde{{\mathcal{O}}}(d^4)$. Explicit choices of_$L$_and $T$ are given in the proof_of Proposition \[prop:policy\_evaluation\_complexity\], which_may_be found in Appendix_\[app:complexity\].

Entropic Policy Improvement {#sec:entropic_regularization}
---------------------------

Next we_describe a means of approximating Algorithm_\[alg:exact\_pi\]. Note_that since_the gain vector for any element of ${\Pi_\eta(P,Q)}$ is constant, we_need only improve policies with respect