p-4.4\] for constrained MDPs). Our results for unconstrained (resp., constrained) MDPs given in this paper can be compared with some of the prior results in [@HL99 Chap. 12] and [@HL02] (resp., [@HGL03] and [@KNH00]) for lower-semicontinuous models.

While this paper focuses on the average cost criterion, the analysis we give, with minor changes, can also be applied to discounted-cost, constrained or multi-objective MDPs similar to those studied in [@FeS96; @HG00; @HR04], for finding constrained optimal or Pareto optimal policies (for a given initial distribution) using the LP approach, in the case of countable action spaces. We also mention that based on similar ideas, we introduced another majorization condition for MDPs where both the state and action spaces are Borel, in a separate recent work [@Yu20]; there we used the majorization condition instead of the commonly required continuity/compactness conditions to prove the average cost optimality inequalities via the vanishing discount factor approach.

The rest of this paper is organized as follows. Section \[sec-2\] gives background materials about the average-cost MDP model, some prior optimality results for the minimum pair approach, and an overview of linear programs in topological vector spaces. Section \[sec-3\] presents our LP formulation and duality results for unconstrained MDPs, and Section \[sec-4\] extends the results to constrained MDPs. Proofs for the theorems in Sections \[sec-3\] and \[sec-4\] are given in Section \[sec-5\].

Preliminaries {#sec-2}
=============

We start with some notations and basic definitions. For a topological space $X$, $\B(X)$ denotes the Borel $\sigma$-algebra on $X$, and $\P(X)$ denotes the set of probability measures on $\B(X)$. Nonnegative or signed measures on $\B(X)$ will be called Borel measures. A *Borel space* (a.k.a. *standard Borel space*) is a separable metrizable space that is homeomorphic to a Borel subset of some Polish space (i.e., a separable and completely metrizable space) [@bs Chap. 7]. Let $X$ and $Y$ be Borel spaces. A *Borel measurable stochastic kernel on $Y
p-4.4\ ] for constrained MDPs). Our results for unconstrained (resp. , constrained) MDPs given in this paper can be compare with some of the anterior results in [ @HL99 Chap.   12 ] and [ @HL02 ] (resp. , [ @HGL03 ] and [ @KNH00 ]) for lower - semicontinuous model. 

 While this newspaper focuses on the average price standard, the analysis we give, with minor changes, can also be applied to discount - cost, constrained or multi - objective MDPs similar to those study in   [ @FeS96; @HG00; @HR04 ], for line up constrained optimal or Pareto optimal policies (for a given initial distribution) use the LP approach, in the case of countable action outer space. We also mention that based on similar idea, we introduced another majorization condition for MDPs where both the state and action spaces are Borel, in a freestanding recent work [ @Yu20 ]; there we used the majorization condition instead of the commonly required continuity / compactness conditions to prove the average cost optimality inequalities via the vanishing discount factor approach. 

 The remainder of this paper is organized as stick to. incision   \[sec-2\ ] gives background materials about the modal - cost MDP model, some prior optimality results for the minimal pair approach, and an overview of linear programs in topological vector spaces. Section   \[sec-3\ ] presents our LP formulation and duality results for unconstrained MDPs, and Section   \[sec-4\ ] extends the results to constrained MDPs. proof for the theorem in Sections   \[sec-3\ ] and   \[sec-4\ ] are given in Section   \[sec-5\ ]. 

 Preliminaries { # sec-2 } 
 = = = = = = = = = = = = = 

 We start with some notations and basic definitions. For a topological space $ X$, $ \B(X)$ announce the Borel $ \sigma$-algebra on $ X$, and $ \P(X)$ denotes the set of probability measures on $ \B(X)$. Nonnegative or sign measures on $ \B(X)$ will be called Borel measures. A * Borel space * (a.k.a.   * standard Borel distance *) is a separable metrizable space that is homeomorphic to a Borel subset of some Polish outer space (i.e., a separable and wholly metrizable space) [ @bs Chap.   7 ]. Let $ X$ and $ Y$ be Borel spaces. A * Borel measurable stochastic kernel on $ Y
p-4.4\] flr constrained MDPs). Our vesults for uncouwtrainxd (resp., constrakned) MDPs given in this papec cab be xompared with some of ghe prior results in [@IL99 Chap. 12] and [@HL02] (rxap., [@HGL03] and [@KNG00]) for oower-semicontikuous modelv.

While this pa[ef yocuses on the average cost criteriog, the amapysis we give, rith iinod changes, can also be applied to djscounttd-cost, constrained or multi-objective MDPs sililag to those studied in [@FeS96; @HG00; @HT04], fow finding conrtrained oktnmal or Pargto optimal policies (for a given initnal distribocuon) osing the LP apprjach, in the gsse of countanle action spages. Wx alwo mention that based on similar ideas, re introdgczd another majorizatiin condhtiot fof MDOs shxre both hhe state and action spaxes are Borel, in a xe[qrate recent sork [@Yt20]; ehere we used the majorization conditiot ihstead of the commonly eequired continuity/colpactness conditions to prove the average cost optimality hnequelktits via gye vanishing discount factor approach.

The rest os tnix paper is orgcnized as followx. Devjion \[sec-2\] gives cackgrouns materials about hhe avetage-cowt MDP moqel, xome prior optimality resulrs for the mpnimym pair approach, aud an overvizw of kineat programs in topologiccl vecfor spaces. Dection \[sed-3\] presents our LP fogmuldtion and duality results sor unconwtranned MDPr, anc Sectyon \[sec-4\] extfnds bve results to conshraingd MDPv. Proofs flr the theorems in Sections \[sec-3\] eid \[sec-4\] are givgn hn Vection \[szc-5\].

Prellminaries {#sec-2}
=============

We start with soke notacions xnd basic sefinitmons. For a tjpological spdfe $X$, $\B(X)$ denoves the Bjrel $\signa$-algebfx on $X$, and $\P(X)$ cenotes tkt set of peobability measurex ov $\B(X)$. Nonnegative or signed measurex ov $\B(V)$ aikl te called Bosel oearires. X *Borel spage* (x.k.a. *syandard Borel space*) hs a separable metrizanlc space tyat is hjmeomorphic tp a Borel subset ov somx Polirh spsce (i.e., a separable and completely metrizabpe fpace) [@bs Cha[. 7]. Leb $X$ wnd $Y$ be Bprel spaces. A *Borel measurable stochastmc kernel on $Y
p-4.4\] for constrained MDPs). Our results for constrained) given in paper can be prior in [@HL99 Chap. and [@HL02] (resp., and [@KNH00]) for lower-semicontinuous models. While paper focuses on the average cost criterion, the analysis we give, with minor can also be applied to discounted-cost, constrained or multi-objective MDPs similar to those in @HG00; for constrained optimal or Pareto optimal policies (for a given initial distribution) using the LP approach, in case of countable action spaces. We also mention based on similar ideas, introduced another majorization condition for where the state action are in a separate work [@Yu20]; there we used the majorization condition instead of the commonly required continuity/compactness conditions to prove average cost via the discount approach. rest of this organized as follows. Section \[sec-2\] gives the average-cost MDP model, some prior optimality results the minimum approach, and an overview of linear in topological vector spaces. Section \[sec-3\] presents our formulation and duality results for unconstrained MDPs, and Section \[sec-4\] extends the results to constrained for the theorems in \[sec-3\] and \[sec-4\] given Section Preliminaries ============= We with some notations and basic definitions. For a topological space $X$, denotes the Borel $\sigma$-algebra on $X$, and $\P(X)$ denotes the probability on $\B(X)$. Nonnegative signed measures on $\B(X)$ be Borel measures. A *Borel *standard space*) metrizable that homeomorphic to a Borel of some Polish space (i.e., separable and completely metrizable $X$ and $Y$ be Borel spaces. A *Borel stochastic kernel on $Y
p-4.4\] for constrained MDPs). Our resUlts for uncOnstrAinEd (rEsP., conStraIned) MDPs given iN This Paper can be compared with Some oF tHE priOR rEsultS in [@HL99 ChAP. 12] aND [@hL02] (rEsP., [@HgL03] aNd [@knH00]) For loWer-SemiconTinuous modEls.

whIle this paper FOcUses on the aVerAge cost criteRioN, the anAlYsiS We givE, wiTh minOr chanGEs, can aLso be applIeD To discOUnted-coST, CoNstrAined or multi-objecTIvE mDPs similar to tHose stUdIEd IN [@feS96; @hG00; @Hr04], for findinG cOnstrAIned optIMaL OR parETo optimal poliCies (for a givEN inItial dIsTriBUtion) uSing tHe lp apProach, in the Case Of countabLe actiON spaces. wE also meNtion tHat BasEd on SImIlAr iDeAS, we INtRodUCed Another mAjOrIzatiOn coNDITIon fOr MdPs wHere bOth the state anD acTion SPacEs are borel, In a sEpArate Recent Work [@YU20]; tHere we used the maJoriZation conDitIoN inStEad of THe commOnlY reQuired cOntinuiTY/coMpACTNeSs conditions to provE tHE AvErage cosT optimALiTy INequalitIeS viA the VANishiNg diSCoUnt factoR approACh.

thE rest of ThIs papeR iS orGanIzed aS FollOws. SecTion \[sec-2\] gIves bACkground materiALs about the aveRAgE-COsT mDP mOdeL, some prior oPtimALity ResuLTs For THe minImum pAiR ApPRoach, and an overview oF lInear pRograMs in topologicAl vector spACES. Section \[Sec-3\] pREsENts our LP formulAtion And duality REsults foR uncoNstraineD MDPs, and SECTion \[sec-4\] eXteNds The ResULTs To constrained mdps. PrOoFs for thE thEorems iN SeCtiOns \[Sec-3\] AnD \[sec-4\] are giVen in SecTiOn \[SeC-5\].

PRelIminaRIes {#sec-2}
=============

We StArt WiTh sOme noTAtions And baSic dEfInITioNs. For a tOPoLOGicaL sPaCe $X$, $\B(x)$ deNoTes thE BorEL $\siGma$-algeBra on $X$, and $\p(X)$ dENoteS tHe Set of prObability measUrEs on $\B(X)$. NonnEgAtiVe or siGNEd measurEs on $\B(X)$ will be called Borel MEasures. a *BoRel spAce* (a.K.a. *standarD BoRel spaCe*) iS A separAble meTrizaBlE spACE that IS HoMeoMoRphic to a BoREL suBset oF sOme POlish spAce (i.e., a separable and COmpLetely metrizaBle SpacE) [@BS CHap. 7]. lEt $x$ And $y$ bE borEL Spaces. A *Borel meaSurable stoChAStIc kernel on $y
p-4.4\] for constrained MD Ps). Our r esult s f orun cons trai ned (resp., co n stra ined) MDPs given in th is pa pe r  can be  comp ared wi t hs o meof  t hepr i or  resu lts  in [@H L99 Chap.12]  a nd [@HL02] ( r es p., [@HGL0 3]and [@KNH00] ) f or low er -se m icont inu ous m odels. 
While  this pap er focuse s  on the a ve rage  cost criterion,t he analysis we gi ve, wi th mi n o r c han ges, can a ls o bea ppliedt od i s cou n ted-cost, con strained or mul ti-obj ec tiv e  MDPssimil ar tothose studi ed i n [@FeS96 ; @HG0 0 ; @HR04 ] , for f inding  co nst rain e dop tim al orP ar eto opt imal pol ic ie s (fo r ag i v e n in iti al d istri bution) using  th e LP app roach , intheca se of  count ableac tion spaces. We  als o mention  th at  ba se d ons imilar  id eas , we in troduce d  an ot h e r  m ajorization condit io n fo r MDPs w here b o th  t h e statean d a ctio n space s ar e  B orel, in  a sep a ra te  recent  w ork [@ Yu 20] ; t herew e us ed the  majoriz ation condition inst e ad of the com m on l y  r e quir edcontinuity/ comp a ctne ss c o nd iti o ns to  prov et he average cost optima li ty ine quali ties via thevanishingd i s count fa ctor ap p roach.

The re st of  this pape r  is orga nized  as foll ows. Sect i o n \[sec- 2\]  gi ves  ba c k gr ound material s abou tthe ave rag e-costMDP  mo del , s om e prior o ptimalit yre su lt s f or th e  minimum  p air  a ppr oach, and an  over view  o fl ine ar prog r am s in t op ol ogic alve ctorspac e s.Section  \[sec-3\ ] p r esen ts  o ur LP f ormulation an dduality re su lts  for u n c onstrain ed MDPs, and Section \[ s ec-4\]ext endstheresults t o c onstra ine d  MDPs.  Proof s for  t het h eorem s in  Se ct ions \[sec - 3 \]and \ [s ec-4 \] aregiven in Section \ [ sec -5\].

Prelim ina ries { #s ec- 2 }= === == = === = = =

We start wit h some not at i on s and basi c  de fi nitions . For a  topo l ogicalspace $X$ , $\B(X)$  d enot e s  th e Borel $\ sigma$-a lgebra on $X$,a nd  $\P( X)$  denot es  th e set  of pr o bab ility  measu re s on $ \B(X) $.  Nonnega tive or signed measures  on $\ B(X)$  wi ll be cal led Bor el measur es.A *Borel s pac e*(a.k. a.* stand ardB or els pace* ) is a separab l emet r i za ble space t h a t  is  home omo r phic t o aBorel subset of s o me Polish spac e (i . e .,a s e para bl e and complete lyme t r izable s pa ce) [@bs Ch ap. 7].Le t  $X$and $Y $ be B orel sp a c es .  A *Bo relmea surable s toc ha s tic ker ne lo n $Y
p-4.4\] for_constrained MDPs)._Our results for unconstrained_(resp., constrained)_MDPs_given in_this_paper can be_compared with some_of the prior results_in [@HL99 Chap. 12]_and_[@HL02] (resp., [@HGL03] and [@KNH00]) for lower-semicontinuous models.

While this paper focuses on the average_cost_criterion, the_analysis_we_give, with minor changes, can_also be applied to discounted-cost,_constrained or_multi-objective MDPs similar to those studied in [@FeS96; @HG00;_@HR04],_for finding constrained_optimal or Pareto optimal policies (for a given initial_distribution) using the LP approach, in_the case of_countable_action_spaces. We also mention_that based on similar ideas, we_introduced another majorization condition for MDPs_where both the state and action spaces_are Borel, in a separate recent_work [@Yu20]; there we used_the majorization_condition instead of the commonly_required continuity/compactness conditions_to prove_the average cost_optimality inequalities via the vanishing discount_factor approach.

The rest_of this paper is organized as_follows._Section \[sec-2\] gives background_materials_about_the average-cost_MDP model, some_prior_optimality results_for_the minimum pair approach, and an_overview_of linear programs in topological vector spaces._Section \[sec-3\] presents our LP_formulation_and duality results for_unconstrained MDPs, and Section \[sec-4\] extends_the results to constrained MDPs. Proofs_for the_theorems in_Sections \[sec-3\] and \[sec-4\] are given in Section \[sec-5\].

Preliminaries {#sec-2}
=============

We start with some notations_and basic definitions. For a topological_space $X$, $\B(X)$ denotes_the Borel_$\sigma$-algebra_on $X$, and_$\P(X)$_denotes the_set of probability measures on $\B(X)$. Nonnegative_or signed_measures on $\B(X)$ will be called_Borel measures. A *Borel_space*_(a.k.a. *standard Borel space*) is a separable_metrizable space that is homeomorphic to_a Borel subset of some_Polish_space_(i.e., a separable and completely_metrizable space) [@bs Chap. 7]. Let $X$_and $Y$ be_Borel spaces. A *Borel measurable stochastic kernel_on_$Y