title: How to implement q-learning in R?
body: <p>I am learning about q-learning and found a Wikipedia post and this <a href="http://mnemstudio.org/path-finding-q-learning-tutorial.htm" rel="noreferrer">website</a>.</p>
  
  <p>According to the tutorials and pseudo code I wrote this much in R</p>
  
  <pre><code>#q-learning example
  #http://mnemstudio.org/path-finding-q-learning-tutorial.htm
  #https://en.wikipedia.org/wiki/Q-learning
  
  set.seed(2016)
  iter=100
  dimension=5;
  alpha=0.1 #learning rate
  gamma=0.8 #exploration/ discount factor
  
  # n x n matrix
  Q=matrix( rep( 0, len=dimension*dimension), nrow = dimension)
  Q
  
  # R -1 is fire pit,0 safe path and 100 Goal state########
  R=matrix( sample( -1:0, dimension*dimension,replace=T,prob=c(1,2)), nrow = dimension)
  R[dimension,dimension]=100
  R #reward matrix
  ################
  
  for(i in 1:iter){
    row=sample(1:dimension,1)
    col=sample(1:dimension,1)
    I=Q[row,col] #randomly choosing initial state
  
    Q[row,col]=Q[row,col]+alpha*(R[row,col]+gamma*max(Qdash-Q[row,col])
    #equation from wikipedia
  
  }
  </code></pre>
  
  <p>But I have problem in <code>max(Qdash-Q[row,col]</code> which according to the website is <code>Max[Q(next state, all actions)]</code>  How to I programmatically search all actions for next state?</p>
  
  <p>The second problem is this pseudo code</p>
  
  <pre><code>Do While the goal state hasn't been reached.
  
  Select one among all possible actions for the current state.
  Using this possible action, consider going to the next state.
  Get maximum Q value for this next state based on all possible actions.
  Compute: Q(state, action) = R(state, action) + Gamma * Max[Q(next state, all actions)]
  Set the next state as the current state.
  End Do
  </code></pre>
  
  <p>Is it this</p>
  
  <pre><code>while(Q&lt;100){
  Q[row,col]=Q[row,col]+alpha*(R[row,col]+gamma*max(Qdash-Q[row,col])
  }
  </code></pre>