time Through exploration, despite the initial (patient) action resulting in a larger cost (or negative reward) than in the forceful strategy, the overall cost is lower, thus revealing a more rewarding strategy. Algorithm After steps into the future the agent will decide some next step. The weight for this step is calculated as , where (the discount factor) is a number between 0 and 1 () and has the effect of valuing rewards received earlier higher than those received later (reflecting the value of a "good start"). may also be interpreted as the probability to succeed (or survive) at every step . The algorithm, therefore, has a function that calculates the quality of a state–action combination: . Before learning begins, is initialized to a possibly arbitrary fixed value (chosen by the programmer). Then, at each time the agent selects an action , observes a reward , enters a new state (that may depend on both the previous state and the selected action), and is updated. The core of the algorithm is a Bellman equation as a simple value iteration update, using the weighted average of the current value and the new information: where is the reward received when moving from the state to the state , and is the learning rate . Note that is the sum of three factors: : the current value (weighted by one minus the learning rate) : the reward to obtain if action is taken when in state (weighted by learning rate) : the maximum reward that can be obtained from state (weighted by learning rate and discount factor) An episode of the algorithm ends when state is a final or terminal state. However, Q-learning can also learn in non-episodic tasks (as a result of the property of convergent infinite series). If the discount factor is lower than 1, the action values are finite even if the problem can contain infinite loops. For all final states , is never updated, but is set to the reward value observed for state . In most cases, can be taken to equal zero. Influence of variables Learning rate The learning rate or step size determines to what extent newly acquired information overrides old information. A factor of 0 makes the agent learn nothing (exclusively exploiting prior knowledge), while a factor of 1 makes the agent consider only the most recent information (ignoring prior knowledge to explore possibilities). In