 To get around these limitations, new, separate options must be defined, increasing the problem’s branching factor, and care must be taken to avoid loops (if so desired). An MMDP coarse action leaves the “direction” of the action undecided: the same fine policy may be executed starting in several bottleneck states, and may take the agent in one of several directions until arriving at one of multiple destinations from which different successor coarse actions may be taken. In the context of MMDPs, if one wanted to be able to transfer policies on the same cluster which guide an agent in different particular directions, separate local policies would need to be stored in the “database” of solved tasks. However, we would only need to transfer and plan with [*one*]{} of them.

-   A strength of the options framework is that multiple related queries, or tasks, may be solved essentially within the same SMDP. However, the tasks must be closely related in specific ways (e.g. tasks differing only in the goal state), and this strength comes at the expense of ignoring problem-specific information when one only wants to solve one problem. Our approach to the construction of MMDPs differs in that while we assume a particular problem when building a decomposition, we are able to consider a broader set of transfer possibilities.

-   Bottlenecks and partitioning do not explicitly enter the picture in options or SMDPs. Options may be defined on any subset of the statespace, and in applications may often take the form of a macro-action which directs the agent to an intermediate goal state starting from [*any*]{} state in a (possibly large) neighborhood. For example, an option may direct a robot to a hallway from any state in a room. We constrain our “initiation” and “termination” sets to be bottleneck states, however this means that learning policies at coarse scales is fast, and can be carried out completely independent of other scales. Coarse scale learning involves only the bottleneck states, giving a drastically reduced computational complexity. Provided the partitioning of a scale is well chosen, this construction allows one to capitalize on improved mixing times to accelerate convergence.

-   MMDPs are a representation for MDPs: we cannot solve problems that cannot be phrased as an MDP (i.e. problems whose solutions require non-Markov policies). A policy solving an MMDP, at any scale, is a Markov policy. SMDPs may in general have non-
To get around these limitations, new, separate choice must be define, increasing the problem ’s branching divisor, and care must be taken to debar loops (if so desire). An MMDP coarse action leaves the “ commission ” of the action undecided: the like fine policy may be executed starting in several bottleneck states, and may take the agentive role in one of several directions until arriving at one of multiple destination from which different successor coarse actions may be take. In the context of MMDPs, if one desire to be able to transfer policies on the same cluster which guide an agentive role in different particular directions, separate local policies would need to be store in the “ database ” of solved tasks. However, we would only need to transfer and plan with [ * one * ] { } of them. 

 -    A strength of the options framework is that multiple related queries, or undertaking, may be solved essentially within the like SMDP. However, the undertaking must be close related in specific ways (e.g. tasks differing only in the goal state), and this lastingness comes at the expense of ignoring problem - specific information when one only wants to clear one problem. Our approach to the construction of MMDPs differs in that while we wear a particular trouble when build a decomposition, we are able to consider a broader hardening of transfer possibilities. 

 -    Bottlenecks and partitioning do not explicitly record the picture in options or SMDPs. Options may be defined on any subset of the statespace, and in application may often consume the form of a macro - military action which directs the agentive role to an intermediate goal state starting from [ * any * ] { } department of state in a (possibly large) neighborhood. For exercise, an option may address a robot to a hallway from any state in a room. We constrain our “ initiation ” and “ termination ” sets to be bottleneck states, however this mean that learning policies at coarse scale is fast, and can be carried out completely independent of other scales. Coarse plate learning involves only the bottleneck states, giving a drastically reduced computational complexity. Provided the breakdown of a plate is well choose, this construction allows one to capitalize on improved mixing times to accelerate overlap. 

 -    MMDPs are a representation for MDPs: we cannot solve problems that cannot be phrased as an MDP (i.e. problems whose solutions require non - Markov policies). A policy solving an MMDP, at any scale, is a Markov policy. SMDPs may in cosmopolitan have non-
 To get around these limitauions, new, separatg iptionv must be defived, increasing the problem’s uranxhing factor, and care must ce taken no avoid ooopw (if so desmded). An MMDP ckwrse ection leaves tme “directiot” of the actiot jnbecided: the same fine policy may be qxecutec dtarting in sederak botfleneck states, and may take the agsnt in mne of severak directions until arrivinh at one of multiple dfstinations froi which diffefent successor coarse zctions may be taken. In the congext pf MMDPs, id ine fanted to bx able to transfer policiev on thr same cluster whmch tuide an agent in difherent particular ditections, sapcrate local policies qoyld nged tm be wtofed ii tge “datwbaae” of solvsd tasks. Hoqever, we would only nqvc to transfed and [lwn with [*one*]{} of them.

-   A strength of the opfions framework is that multiple related queties, or tafks, may be solved essentially within the same SMD[. Howxvdr, uhc gqsns must be closely related in specific ways (e.d. tssls differing okly in the goal ststf), sgd this streneth comes at the expense of ignorigg priblem-specyfic information when one only qants to solne obe problem. Our appxoach to the conxtrucyion of MMDPs differs iu that while we adsume a pzfticular problem whvn bgilding a decomposition, we are able to eonsider a btoader fet of trajsfer possibilities.

-   Bohtlengcks atd partitilning do not explicitly enter tix picture in ppdiots or SMBPs. Opbions may be desined on any sobset of che stxtespace, ahd in a'plications iay often taka the form of a macro-wctiin wyich difdcts the agent to an innexmediate toal state startinn froo [*any*]{} state in a (powsibly large) neoghcorrold. Fjs example, an optkon kay dkrect a robig to a hallway from any vtats in a room. We conxtvain our “unitiatijn” and “terminstion” sets to be blttleieck svates, nowgver this means that learning pklicies ah cjarse scales is nast, and can bz carried out completely independent of ither scales. Coarsw scale learning inrokves only tie boteleneck sdates, giving a drastucally reduced cokputational complexitg. Provhded hhe partitioning of a scale is well chosen, this construction allows one ti capivajize on impdovec mixpng cimes tj acreuerate convergenbe.

-   MMDPs are a representation hor MDPs: wa eannot solve problems that camnut be phrased as an MDP (i.e. problems wgose solitions require non-Markov policirs). A policy solving an MKDP, at aby sczle, is a Merkov policy. SMDPs msy in yenerao have bon-
To get around these limitations, new, separate be increasing the branching factor, and avoid (if so desired). MMDP coarse action the “direction” of the action undecided: same fine policy may be executed starting in several bottleneck states, and may the agent in one of several directions until arriving at one of multiple from different coarse may be taken. In the context of MMDPs, if one wanted to be able to transfer on the same cluster which guide an agent different particular directions, separate policies would need to be in “database” of tasks. we only need to and plan with [*one*]{} of them. - A strength of the options framework is that multiple related or tasks, solved essentially the SMDP. the tasks must related in specific ways (e.g. tasks the goal state), and this strength comes at expense of problem-specific information when one only wants solve one problem. Our approach to the construction MMDPs differs in that while we assume a particular problem when building a decomposition, we to consider a broader of transfer possibilities. Bottlenecks partitioning not enter the in options or SMDPs. Options may be defined on any subset the statespace, and in applications may often take the form macro-action directs the agent an intermediate goal state from state in a (possibly For an a to hallway from any state a room. We constrain our and “termination” sets to means that learning policies at coarse scales is and can be carried out completely independent other scales. Coarse scale learning involves only the bottleneck states, giving a reduced computational the partitioning of a scale is well chosen, construction allows one to on improved mixing times to accelerate convergence. - MMDPs a for MDPs: cannot solve problems cannot be phrased an MDP (i.e. solutions require policies). policy at any scale, is a Markov SMDPs in general have non-
 To get around these limitatioNs, new, separAte opTioNs mUsT be dEfinEd, increasing thE ProbLem’s branching factor, and Care mUsT Be taKEn To avoId loops (IF sO DEsiReD). AN MMdP COaRse acTioN leaves The “directiOn” oF tHe action undeCIdEd: the same fIne Policy may be eXecUted stArTinG In sevEraL bottLeneck STates, aNd may take ThE Agent iN One of seVERaL dirEctions until arrivINg AT one of multiple DestinAtIOnS FRom WhiCh differenT sUccesSOr coarsE AcTIONs mAY be taken. In the Context of MMdps, iF one waNtEd tO Be able To traNsFEr pOlicies on thE samE cluster wHich guIDe an ageNT in diffErent pArtIcuLar dIReCtIonS, sEParATe LocAL poLicies woUlD nEed to Be stORED In thE “daTabaSe” of sOlved tasks. HowEveR, we wOUld Only nEed to TranSfEr and Plan wiTh [*one*]{} Of Them.

-   A strength of The oPtions fraMewOrK is ThAt mulTIple reLatEd qUeries, oR tasks, mAY be SoLVED eSsentially within thE sAME SmDP. HowevEr, the tASkS mUSt be closElY reLateD IN specIfic WAyS (e.g. tasks DifferINg OnLy in the GoAl statE), aNd tHis StrenGTh coMes at tHe expensE of igNOring problem-spECific informatIOn WHEn ONe onLy wAnts to solve One pRObleM. Our APpRoaCH to thE consTrUCtIOn of MMDPs differs in tHaT while We assUme a particulaR problem whEN BUilding a DecoMPoSItion, we are able To conSider a broaDEr set of tRansfEr possibIlities.

-   BoTTLenecks aNd pArtItiOniNG Do Not explicitly ENTer tHe Picture In oPtions oR SMdPs. optIonS mAy be definEd on any sUbSeT oF tHe sTatesPAce, and in ApPliCaTioNs may OFten taKe the Form Of A mACro-Action wHIcH DIrecTs ThE ageNt tO aN inteRmedIAte Goal staTe startinG frOM [*any*]{} StAtE in a (posSibly large) neiGhBorhood. For ExAmpLe, an opTIOn may dirEct a robot to a hallway from ANy state In a Room. WE conStrain our “IniTiatioN” anD “TerminAtion” sEts to Be BotTLEneck STAtEs, hOwEver this meANS thAt leaRnIng pOlicies At coarse scales is faST, anD can be carried Out CompLETeLy iNDePEndEnT Of oTHEr scales. Coarse sCale learniNg INvOlves only tHE boTtLeneck sTates, giVing a DRasticaLly reduceD computatIoNal cOMPleXity. ProvidEd the parTitioning OF a scaLE iS well ChoSen, thiS cOnsTructIon allOWs oNe to cApitalIzE on impRoved MiXing timeS to accelerate convergencE.

-   MMDPs Are a rEprEsentatioN foR mDPS: we cannot SolvE problems tHat CanNot be PhrASed as An MDp (I.e. ProBLems wHose SOlutions rEQuIre NON-MArkov policiES). a PolIcy soLviNG an MMDp, at aNy scale, is a Markov pOLicy. SMDPs may in GeneRAL haVe nON-
 To get around these limit ations, ne w, se par ate  o ptio ns m ust be defined ,  inc reasing the problem’sbranc hi n g fa c to r, an d carem us t beta ke n t oa vo id lo ops  (if so  desired).  An  M MDP coarse a c ti on leavesthe  “direction”  of  the a ct ion undec ide d: th e same fine p olicy may  b e  execu t ed star t i ng  inseveral bottlenec k  s t ates, and maytake t he ag e n t i n o ne of seve ra l dir e ctionsu nt i l arr i ving at one o f multipled est inatio ns  fr o m whic h dif fe r ent  successorcoar se action s mayb e taken .  In the  conte xtofMMDP s ,if  on ew ant e dtob e a ble to t ra ns fer p olic i e s on t hesame  clus ter which gui dean a g ent  in d iffer entpa rticu lar di recti on s, separate loc al p olicies w oul dnee dto be stored  in  th e “data base” o f  so lv e d ta sks. However, we w ou l d  o nly need  to tr a ns fe r  and pla nwit h [* o n e*]{}  oft he m.

-  A stre n gt hof theop tionsfr ame wor k ist hatmultip le relat ed qu e ries, or tasks ,  may be solve d  e s s en t iall y w ithin the s ameS MDP.  How e ve r,t he ta sks m us t  b e  closely related in  s pecifi c way s (e.g. tasks  differing o n ly in th e go a ls tate), and thi s str ength come s  at theexpen se of ig noring pr o b lem-spec ifi c i nfo rma t i on  when one onl y want sto solv e o ne prob lem . O urapp ro ach to th e constr uc ti on  o f M MDPsd iffers i ntha twhi le we assume  a pa rtic ul ar pro blem wh e nb u ildi ng  a  dec omp os ition , we are  able t o conside r a broa de rset oftransfer poss ib ilities.
-  B ottlen e c ks and p artitioning do not expl i citly e nte r the  pic ture in o pti ons or  SM D Ps. Op tionsmay b edef i n ed on a ny  su bs et of thes t ate space ,andin appl ications may often tak e the form of  amacr o - ac tio n  w h ich  d i rec t s  the agent to a n intermed ia t egoal state sta rt ing fro m [*any *]{}s tate in  a (possi bly large )neig h b orh ood. For e xample,an option may d i re ct arob ot toahal lwayfrom a n y s tatein a r oo m. Weconst ra in our “ initiation” and “termin ation”  sets  to  be bottl ene c k s tates, ho weve r this mea nstha t lea rni n g pol icie s  a t c o arsescal e s is fast ,  a ndc a nbe carriedo u t  co mplet ely indepe nden t of other scales .  Coarse scalelear n i nginv o lves  o nly the bottle nec ks t ates, gi vi ng a drasti cally re du c ed co mputat ionalcomplex i t y. Provid ed t hepartition ing  o f  a scal eis well c hose n,  thisconstr u ctio n allows one to ca pital i z e oni mpr ovedmi xing ti m es t o accelera te converge nce.
-  MMDPs  are are presen tat io n for MDPs :  we canno t sol ve prob le ms t hat  canno t be p hrase d as  a n M DP (i.e.p r ob l em sw hos e so lutio ns  req uire non- M arkov po lic i es). Apo lic y solvin g  a n MMDP, at a nyscale , is a Marko v  pol i cy .  SMDP s mayin gen eral ha v e n on -
 To_get around_these limitations, new, separate_options must_be_defined, increasing_the_problem’s branching factor,_and care must_be taken to avoid_loops (if so_desired)._An MMDP coarse action leaves the “direction” of the action undecided: the same fine_policy_may be_executed_starting_in several bottleneck states, and_may take the agent in_one of_several directions until arriving at one of multiple_destinations_from which different_successor coarse actions may be taken. In the context_of MMDPs, if one wanted to_be able to_transfer_policies_on the same cluster_which guide an agent in different_particular directions, separate local policies would_need to be stored in the “database”_of solved tasks. However, we would_only need to transfer and_plan with_[*one*]{} of them.

-  _A strength of_the options_framework is that_multiple related queries, or tasks, may_be solved essentially_within the same SMDP. However, the_tasks_must be closely_related_in_specific ways_(e.g. tasks differing_only_in the_goal_state), and this strength comes at_the_expense of ignoring problem-specific information when one_only wants to solve_one_problem. Our approach to_the construction of MMDPs differs_in that while we assume a_particular problem_when building_a decomposition, we are able to consider a broader set of_transfer possibilities.

-   Bottlenecks and_partitioning do not explicitly_enter the_picture_in options or_SMDPs._Options may_be defined on any subset of the_statespace, and_in applications may often take the_form of a macro-action_which_directs the agent to an intermediate_goal state starting from [*any*]{} state_in a (possibly large) neighborhood._For_example,_an option may direct a_robot to a hallway from any_state in a_room. We constrain our “initiation” and “termination”_sets_to be bottleneck states, however this_means_that learning policies at coarse scales_is_fast,_and can be carried out_completely independent of other scales. Coarse_scale learning involves only the bottleneck states, giving a_drastically reduced computational_complexity. Provided the partitioning of_a_scale_is well chosen, this construction allows one to capitalize on_improved mixing_times to accelerate_convergence.

-   MMDPs are a representation for MDPs: we_cannot solve problems that cannot be phrased_as an MDP (i.e. problems whose solutions require non-Markov policies). A_policy solving an MMDP, at any scale, is_a Markov policy. SMDPs may in general_have non-