 get the best global maxima. Maybe
algorithms that can encapsulate that will be useful in getting corporations
away from the habit of chasing the highest short-term gains at the expense of
long term viability.

~~~
jetrink
Another way to look at it is that these games have less obvious causal
relationships. If the game is Pac-Man, it is immediately obvious that eating a
dot makes the score go up, because the effect always and immediately follows
the cause. The further the effect is separated from the cause, the more
possibilities the machine must entertain. Imagine, in a different game, you
activate three switches and a door opens. Why did the door open? Maybe all
three switches must be activated for the door to open. Maybe the final switch
controls the door and the other switches control something else. Maybe you
must always activate the switches in a particular order as a security measure.
The game probably gives context (e.g. the third switch is labeled 'open door')
that a human can use to eliminate the many possibilities, but the machine must
experiment before knowing what is relevant. When you separate cause and effect
in time, the machine must deal with many possible'switches'.

------
Voloskaya
Go-explore is weird for me. The main reasons for it's success are all based on
very specific thing that barely transfer to anything such as the ability to
save a state and restart from it at anytime (in this case using features from
the emulator). It look almost like an engineering project.

I don't like bad-mouthing the work of others, so I kind of feel bad for saying
that and maybe someone can prove me wrong: But is this not borderline
"cheating" (in quotes because obviously there is no agreed-upon rules)?

The goal never was to solve Montezuma's revenge just for the sake of it. We
could have done that way before by using hard engineering. The interesting
thing about this game is that it reveals inherent flaws in our reinforcement
learning approach. And so, if you manage to solve this game using a domain-
transferable approach, surely that means you have made some significant
progress in RL. That's doesn't seem to be the case here?

~~~
svantana
I thought so too at first, but here's my new thinking:

The most efficient
get the best global maxima. possibly 
 algorithm that can encapsulate that will be utilitarian in getting corporations 
 away from the substance abuse of chasing the highest short - condition gains at the expense of 
 long condition viability. 

 ~~~ 
 jetrink 
 Another way to look at it is that these game have less obvious causal 
 relationships. If the game is Pac - Man, it is immediately obvious that eating a 
 dot makes the mark go up, because the effect always and immediately be 
 the cause. The further the effect is separated from the campaign, the more 
 possibilities the machine must entertain. Imagine, in a different game, you 
 trip three switches and a door opens. Why did the door open? Maybe all 
 three switch must be activated for the door to open. Maybe the final switch 
 controls the door and the other switches operate something else. Maybe you 
 must always trip the switch in a finical order as a security measurement. 
 The plot probably gives context (e.g. the third substitution is labeled' open door') 
 that a human can use to eliminate the many possibility, but the machine must 
 experiment before knowing what is relevant. When you separate cause and consequence 
 in time, the car must share with many possible'switches'. 

 ------ 
 Voloskaya 
 Go - explore is weird for me. The main reasons for it's achiever are all based on 
 very specific thing that barely transfer to anything such as the ability to 
 save a state and restart from it at anytime (in this case using feature from 
 the emulator). It attend almost like an technology project. 

 I don't like bad - mouthing the employment of others, so I kind of palpate bad for saying 
 that and maybe someone can prove me wrong: But is this not boundary line 
 " cheating " (in quotes because obviously there is no agree - upon rules)? 

 The goal never was to solve Montezuma's revenge just for the sake of it. We 
 could have make that way before by use hard engineering. The interesting 
 thing about this game is that it reveals inherent flaw in our reinforcement 
 learning approach. And so, if you manage to solve this game use a domain- 
 movable approach, surely that means you have made some significant 
 progress in RL. That's doesn't seem to be the event here? 

 ~~~ 
 svantana 
 I thought so too at foremost, but here's my new thinking: 

 The about efficient
 geh the best global maxima. Maybe
algorithms that ran enczpsulate that will be useful in gettmng xorpoeations
away from the hxbit of cjasing tye hmghest short-term gains ab the cxpenvx of
long term vlability.

~~~
jetsink
Another waf go look at it is that these games havq less pbgious causal
rejatipgshils. If the game is Pac-Man, it is immsdiatelj obvious that eaying a
dot makes the score ho uo, because the effeft always abd iinediately foulows
the cause. The furjher the effect is separated froo the cause, the mire
omssibilitiew the machine musb enterdain. Imsgine, in a difnerenv gane, you
activate three vwitches and a doot opens. Whf bid the door open? Mayve all
tvree swigxher mhsv bs actigatxd for the soor to opeb. Maybe the final seiebn
controls ths door agd the other switches control something elae. Maybe you
must always activate the switched in a pawticular order as a security measure.
The game probdbly jixes cjvredt (e.g. the third switch is labeled 'open door')
thwf s muman can use to eliminate yhf kwny possibilijies, buc tge machine must
expfriment befoee knowind whst is relevant. When you sepqrate cause cnd effect
in time, the machine murt dgal wiyh many possible'switcher'.

------
Volkskaya
Go-exppore is wskrd for me. The mxin raasons for it's success are all basev on
vzry speckfic thing that barepy transfer to anything duch cs tha ability ho
save a state and restart from it at anytime (it tvis case usinn features from
ehe emulator). Ij look aliost uike an enfineeriig project.

I qon't like bad-kluthing the xork of oeherw, so I kind uf feel bad fot saying
tkct and matbe someone can prpve je wrong: But is thus not borderlime
"cfeaeijg" (ig quotes becagse ubvkpusly there is ni agrred-upon rules)?

The goan neber was to solve Mpnbezuma's rgvenge juft for the sale of it. We
could hwve dmne that wai before by using hard engineerjng. The ijtevesting
thing wboub thys game is that it reveals inherent flaws in our ceinforcement
learning approach. And so, if ypm manage to xolve this gama using a domain-
tranwferable approach, surely that means yoh have made some significant
progress in RL. That's doesn't seem to be the case here?

~~~
svanrana
I vhjught so tok at firsn, bbt kere's mr nex vhinking:

The most vfficient
get the best global maxima. Maybe algorithms encapsulate will be in getting corporations chasing highest short-term gains the expense of term viability. ~~~ jetrink Another way look at it is that these games have less obvious causal relationships. If game is Pac-Man, it is immediately obvious that eating a dot makes the go because effect and immediately follows the cause. The further the effect is separated from the cause, the more the machine must entertain. Imagine, in a different you activate three switches a door opens. Why did door Maybe all switches be for the door open. Maybe the final switch controls the door and the other switches control something else. Maybe you always activate in a order a measure. The game context (e.g. the third switch is that a human can use to eliminate the possibilities, but machine must experiment before knowing what relevant. When you separate cause and effect in the machine must deal with many possible'switches'. ------ Voloskaya Go-explore is weird for me. The for it's success are based on very thing barely to such as ability to save a state and restart from it at anytime this case using features from the emulator). It look almost engineering I don't like the work of others, I of feel bad for and someone wrong: is not borderline "cheating" (in because obviously there is no rules)? The goal never just for the sake of it. We could done that way before by using hard The interesting thing about this game is that it reveals inherent flaws our reinforcement And so, if you manage to solve this using a domain- transferable surely that means you have made some significant progress RL. doesn't seem be the case ~~~ svantana I so too at here's my thinking: most
 get the best global maxima. MayBe
algorithMs thaT caN enCaPsulAte tHat will be usefuL In geTting corporations
away fRom thE hABit oF ChAsing The highESt SHOrt-TeRm GaiNs AT tHe expEnsE of
long Term viabilIty.

~~~
JeTrink
Another WAy To look at it Is tHat these gameS haVe less ObVioUS causAl
rElatiOnshipS. if the gAme is Pac-MAn, IT is immEDiately OBViOus tHat eating a
dot makeS ThE Score go up, becauSe the eFfECt ALWayS anD immediateLy FolloWS
the cauSE. THE FUrtHEr the effect is Separated frOM thE cause, ThE moRE
possiBilitIeS The Machine must EnteRtain. ImagIne, in a DIfferenT Game, you
ActivaTe tHreE swiTChEs And A dOOr oPEnS. WhY Did The door oPeN? MAybe aLl
thREE SWitcHes Must Be actIvated for the dOor To opEN. MaYbe thE finaL swiTcH
contRols thE door AnD the other switchEs coNtrol someThiNg ElsE. MAybe yOU
must aLwaYs aCtivate The switCHes In A PARtIcular order as a secuRiTY MeAsure.
The Game prOBaBlY Gives conTeXt (e.G. the THIrd swItch IS lAbeled 'opEn door')
THaT a Human caN uSe to elImInaTe tHe manY PossIbilitIes, but thE machINe must
experimeNT before knowinG WhAT Is RElevAnt. when you sepaRate CAuse And eFFeCt
iN Time, tHe macHiNE mUSt deal with many possiBlE'switcHes'.

------
VoLoskaya
Go-explOre is weird FOR Me. The maiN reaSOnS For it's success aRe all Based on
verY Specific Thing That bareLy transfeR TO anythinG suCh aS thE abILItY to
save a state AND resTaRt from iT at Anytime (In tHis CasE usInG features From
the eMuLaToR). IT loOk almOSt like an EnGinEeRinG projECt.

I don'T like Bad-mOuThINg tHe work oF OtHERs, so i kInD of fEel BaD for sAyinG
ThaT and mayBe someone Can PRove Me WrOng: But iS this not bordeRlIne
"cheatinG" (iN quOtes beCAUse obvioUsly there is no agreed-upon RUles)?

The GoaL neveR was To solve MoNteZuma's rEveNGe just For the Sake oF iT. We
COUld haVE DoNe tHaT way before BY UsiNg harD eNginEering. THe interesting
thing ABouT this game is thAt iT revEALs InhEReNT flAwS In oUR Reinforcement
leArning apprOaCH. ANd so, if you mANagE tO solve tHis game Using A Domain-
tRansferabLe approacH, sUrelY THat Means you haVe made soMe signifiCAnt
prOGrEss in rL. THat's doEsN't sEem to Be the cASe hEre?

~~~
svAntana
i tHought So too At First, but Here's my new thinking:

The moSt effiCient
 get the best global maxim a. Maybe
a lgori thm s t ha t ca n en capsulate that will  be useful in gettingcorpo ra t ions aw ay fr om theh ab i t  of  c ha sin gt he  high est  short- term gains  at  t he expense o f 
l ong term v iab ility.

~~~jet rink
A no the r  waytolookat iti s that  these ga me s  havel ess obv i o us  cau sal
relationships .  I f  the game is P ac-Man ,i ti s  im med iately obv io us th a t eatin g  a d o t m a kes the score  go up, bec a use  the e ff ect always  andim m edi ately follo ws
t he cause.  The f u rther t h e effec t is s epa rat ed f r om  t heca u se, th e m o repossibil it ie s the  mac h i n e  mus t e nter tain.  Imagine, ina d iffe r ent  game , you 
act iv ate t hree s witch es  and a door ope ns.Why did t hedo orop en? M a ybe al l
t hre e switc hes mus t  be  a c t i va ted for the door t oo p en . Maybethe fi n al  s w itch
con tr ols  the d oor a nd t h eother sw itches co nt rol som et hing e ls e.May be yo u 
mus t alwa ys activ ate t h e switches ina  particular o r de r as a se cur ity measure .
Th e  gam e pr o ba bly gives  cont ex t  ( e .g. the third switc his lab eled'open door')that a hum a n can useto e l im i nate the manypossi bilities,b ut the m achin e must
e xperiment b efore kn owi ngwha t i s re levant. Wheny o u se pa rate ca use  and ef fec t
i n t ime ,the machi ne mustde al  w it h m any p o ssible's wi tch es '.
---- - -
Volo skaya 
Go- ex pl o reis weir d  f o r  me.  T he  mai n r ea sonsfori t's  succes s are all  ba s ed o nve ry spec ific thing th at  barely tr an sfe r to a n y thing su ch as the ability to
sa v e a sta teand r esta rt from i t a t anyt ime (in th is cas e usi ng  fe a t uresf r om 
th eemulator). I t l ook a lm ostlike an  engineering proje c t.
I don't like  ba d-mo u t hi ngt he wor ko f o t h ers, so I kindof feel ba df or  saying
th a t a nd  maybesomeone  canp rove me  wrong: B ut is thi snotb o rde rline
"che ating" ( in quotes becau s eobvio usl y ther eisno ag reed-u p onrules )?

Th egoal n everwa s to sol ve Montezuma's revengejust f or th e s ake of it . W e 
co uld havedone  that waybef ore  by u sin g  hard  eng i ne eri n g. Th e in t erestingt hi nga b ou t this game i s  th at it  re v eals i nher ent flaws in ourr einforcement
l earn i n g a ppr o ach.  A nd so, if youman ag e to solve  t his game us ing a do ma i n-
tr ansfer able a pproach , su r ely th at m ean s you hav e m ad e  some s ig ni f icantprog re ss inRL. Th a t'sd o esn't seem to be  thec a se he r e?
~~~sv antanaI  tho ught so to o at first,  but h ere' s mynew thi nk ing:
The  m ost effici e nt
 get_the best_global maxima. Maybe
algorithms that_can encapsulate_that_will be_useful_in getting corporations
away_from the habit_of chasing the highest_short-term gains at_the_expense of
long term viability.

~~~
jetrink
Another way to look at it is that these games have_less_obvious causal
relationships._If_the_game is Pac-Man, it is_immediately obvious that eating a
dot_makes the_score go up, because the effect always and_immediately_follows
the cause. The_further the effect is separated from the cause, the_more
possibilities the machine must entertain. Imagine,_in a different_game,_you
activate_three switches and a_door opens. Why did the door_open? Maybe all
three switches must be_activated for the door to open. Maybe_the final switch
controls the door and_the other switches control something_else. Maybe_you
must always activate the switches_in a particular_order as_a security measure.
The_game probably gives context (e.g. the_third switch is_labeled 'open door')
that a human can_use_to eliminate the_many_possibilities,_but the_machine must
experiment before_knowing_what is_relevant._When you separate cause and effect
in_time,_the machine must deal with many possible'switches'.

------
Voloskaya
Go-explore_is weird for me._The_main reasons for it's_success are all based on
very_specific thing that barely transfer to_anything such_as the_ability to
save a state and restart from it at anytime (in_this case using features from
the emulator)._It look almost like_an engineering_project.

I_don't like bad-mouthing_the_work of_others, so I kind of feel bad_for saying
that_and maybe someone can prove me_wrong: But is this_not_borderline
"cheating" (in quotes because obviously there_is no agreed-upon rules)?

The goal never_was to solve Montezuma's revenge_just_for_the sake of it. We
could_have done that way before by_using hard engineering._The interesting
thing about this game is that_it_reveals inherent flaws in our reinforcement
learning_approach._And so, if you manage to_solve_this_game using a domain-
transferable approach,_surely that means you have made_some significant
progress in RL. That's doesn't seem to be_the case here?

~~~
svantana
I_thought so too at first,_but_here's_my new thinking:

The most efficient