() to action space (). Optimization objective The goal in a Markov decision process is to find a good "policy" for the decision maker: a function that specifies the action that the decision maker will choose when in state . Once a Markov decision process is combined with a policy in this way, this fixes the action for each state and the resulting combination behaves like a Markov chain (since the action chosen in state is completely determined by and reduces to , a Markov transition matrix). The objective is to choose a policy that will maximize some cumulative function of the random rewards, typically the expected discounted sum over a potentially infinite horizon: (where we choose , i.e. actions given by the policy). And the expectation is taken over where is the discount factor satisfying , which is usually close to 1 (for example, for some discount rate r). A lower discount factor motivates the decision maker to favor taking actions early, rather than postpone them indefinitely. A policy that maximizes the function above is called an optimal policy and is usually denoted . A particular MDP may have multiple distinct optimal policies. Because of the Markov property, it can be shown that the optimal policy is a function of the current state, as assumed above. Simulator models In many cases, it is difficult to represent the transition probability distributions, , explicitly. In such cases, a simulator can be used to model the MDP implicitly by providing samples from the transition distributions. One common form of implicit MDP model is an episodic environment simulator that can be started from an initial state and yields a subsequent state and reward every time it receives an action input. In this manner, trajectories of states, actions, and rewards, often called episodes may be produced. Another form of simulator is a generative model, a single step simulator that can generate samples of the next state and reward given any state and action. (Note that this is a different meaning from the term generative model in the context of statistical classification.) In algorithms that are expressed using pseudocode, is often used to represent a generative model. For example the expression might denote the action of sampling from the generative model where and are the current state and action, and and are the new state and reward. Compared to an episodic simulator, a generative model