we can look at the actions he can takehe can' actually move left at allbut he can move updownor rightand we can assign value to all those actions by going up or rightnothing really happens at allthere' no power pill or dots to consume but if he goes leftthat' definitely negative value we can say for the state given by the current conditions that pac-man is surrounded bymoving down would be really bad choicethere should be negative value for that moving left just can' be done at all moving up or right or staying neutralthe value would remain for those action choices for that given state nowyou can also look ahead little bitto make an even more intelligent agent soi' actually two steps away from getting power pill here soas pac-man were to explore this stateif were to hit the case of eating that power pill on the next statei could actually factor that into the value for the previous state if you just have some sort of discount factorbased on how far away you are in timehow many steps away you areyou can factor that all in together sothat' way of actually building in little bit of memory into the system you can "look aheadmore than one step by using discount factor when computing (here is previous statesis current state) ( , +discount (reward( ,amax( ( ') ( , )sothe value that experience when consume that power pill might actually give boost to the previous values that encountered along the way sothat' way to make qlearning even better the exploration problem one problem that we have in reinforcement learning is the exploration problem how do make sure that efficiently cover all the different states and actions within those states during the exploration phasethe simple approach one simple approach is to always choose the action for given state with the highest value that 've computed so farand if there' tiejust choose at random soinitially all of my values might be and 'll just pick actions at random at first as start to gain information about better values for given actions and given statesi'll start to use those as go butthat ends up being pretty inefficientand can actually miss lot of paths that way if just tie myself into this rigid algorithm of always choosing the best value that 've computed thus far