title: Q-learning in a neural network - Mountain Car
body: <p>So I've been reading about Q-learning and Neural networks. I believe I have the right idea for it however I would like to have a second opinion on my code for NN and updating with Q-values.</p>
  
  <p>I have created a MatLab implementation of the Mountain Car problem and my neural net, I am using neural network toolbox for the NN part. </p>
  
  <p>It is a network of 2 inputs, 5-20 hidden(for experimentation) and 3 outputs(corresponding to actions in mountain car)</p>
  
  <p>the hidden units are set to tansig and the output is purelin and training function is traingdm</p>
  
  <p>Is this the right steps? </p>
  
  <ol>
  <li>obtain inital state s -> [-0.5; 0.0]</li>
  <li>run the network with Qs=net(s) ... this gives me a matrix of 1x3 Q-values corresponding to each action in the intial state s.</li>
  <li>select an action a using e-greedy selection</li>
  <li>simulate the mountain car and obtain s' (the new state as a result of executing action a)</li>
  <li>run the network with Qs_prime=net(s') to get another matrix for the Q-values of s'</li>
  </ol>
  
  <p>Now here is where I  am not sure if this is correct, as I have to figure out how to update the weights for the NN properly.</p>
  
  <ol>
  <li>Compute the QTarget, that is = reward+gamma* Max Q-value from s' ?</li>
  <li>Create a Targets matrix( 1x3) with the Q-values from  the inital s and change the corresponding Q-value for the executed action a to be the QTarget</li>
  <li>use net=Train(net,s,Targets) to update the weights in the NN</li>
  <li>s=s'</li>
  <li>repeat all the above for the new s</li>
  </ol>
  
  <p>Example:</p>
  
  <pre><code>       actions
          1       2      3
  Qs  = 1.3346 -1.9000 0.2371
  
  selected action 3(corresponding to move  mountain car forward)
  
  Qs' = 1.3328 -1.8997 0.2463
  
  QTarget=reward+gamma*max(Qs') = -1+1.0*1.3328 = 0.3328
  
  s= [-5.0; 0.0] and Targets = 1.3346 -1.9000 0.3328
  
  Or I have this wrong and the Targets are 0 0 0.3328 
  
  since we don't know how good the other actions are..
  </code></pre>
  
  <p>here is my Matlab Code( I use R2011 and Neural Network Toolbox)</p>
  
  <pre><code>%create a neural network
  num_hidden=5
  num_actions=3
  net= newff([-1.2 0.6; -0.07 0.07;], [num_hidden,num_actions], {'tansig', 'purelin'},'traingdm');
  
  %network weight and bias initalization
  net= init(net);
  
  %turn off the training window
  net.trainParam.showWindow = false;
  
  %neural network training parameters
  net.trainParam.lr=0.01;
  net.trainParam.mc=0.1;
  net.trainParam.epochs=100
  
  %parameters for q learning
  epsilon=0.9;
  gamma=1.0;
  
  
  %parameters for Mountain car task
  maxEpisodes =10;
  maxSteps=5000;
  reset=false;
  inital_pos=-0.5;
  inital_vel=0.0;
  
  %construct the inital state
  s=[inital_pos;inital_vel];
  Qs=zeros(1,3);
  Qs_prime=zeros(1,3);
  
  %training for maxEpisodes
  for i=1:maxEpisodes
   %each episode is maxSteps long
   for j = 1:maxSteps
  
      %run the network and get Q values for current state Qs-&gt; vector of
      %current Q values for state s at time t Q(s_t)
      Qs=net(s);
  
  
      %select an action
      if (rand() &lt;= epsilon)
          %returns max Q value over all actions
          [Qs_value a]=max(Qs);
      else
          %return a random number between 1 and 3(inclusive)
          a = randint(1,1,3)+1;
      end
  
      %simulate a step of Mountain Car
      [s_prime, action, reward, reset] = SimulateMC(s,a);
  
      %get new Q values for S_prime -&gt; Q(s_t+1)
      Qs_prime=net(s_prime);
  
      %Compute Qtarget for weight updates given by r+y*max Q(s_t+1) over all
      %actions
      Q_target = reward+gamma*max(Qs_prime);
  
      %Create a Targets matrix with the orginal state s q-values 
      Targets=Qs;
  
      %change q-value of the original action to the QTarget
      Targets(a)=Q_target;
  
  
      % update the network for input state s and targets
      [net TR]=train(net,s,Targets);
      %update the state for next step
      s=s_prime;
      %display exactly where the car is to user the NN learns if this output reaches -0.45
      disp(s(1))
  
      if reset==true
          bestSteps=j
          break
      end
   end
   %reset for new episode
   reset=false;
   s=[inital_pos;inital_vel];
  end
  
  %test the network
  %reset state
   s=[inital_pos;inital_vel];
   for i=1:maxEpisodes
      for j=1:maxSteps
          %run the network and get Q values for current state
          Qs=net(s);
  
          %select the max  action always
           [Qs_value a]=max(Qs);
  
          %simulate a step of Mountain Car
          [s_prime, action, reward, reset] = SimulateMC(s,a);
  
          s=s_prime;
          disp(s(1))
      end
       s=[inital_pos;inital_vel];
   end
  </code></pre>
  
  <p>Thanks</p>