title: Optimisation of hill climbing algorithm in c# for training neural networks
body: <p>I have written a small project in C# that creates and trains neural networks. For more details see my previous question here: (<a href="https://scicomp.stackexchange.com/questions/19481">https://scicomp.stackexchange.com/questions/19481</a>).</p>
  
  <p>The neural networks perform well after enough training, but I realise that my self-written hill climbing algorithm may not be perfect and I'm looking for suggestions for improvement. In particular, can I reach the local optimum with less calls to the fitness evaluation function?</p>
  
  <p>There don't seem to be many examples around the web for simple hill climbing algorithms in C#. There is the .NET Math Library, but I would prefer to not have to pay for something.</p>
  
  <p>The hill climbing algorithm runs on every weight and every bias in the network to train the network, and I run multiple passes. I have looked into back propagation, but this seems to be only applied for a single training example, I have ~7000 examples in my training data, and the fitness function evaluates the average performance of the network on all of them and returns a continuous (double) score.</p>
  
  <p>Here is my current code:</p>
  
  <pre><code>    public static double ImproveProperty(ref double property, double startingFitness, int maxIters, Random r, ref Defs.Network network, Func&lt;Defs.Network, double&gt; fitnessFunction)
      {
          //Record starting values
          var lastFitness = startingFitness;
          var lastValue = property;
          //Randomise magnitude of change to reduce chance 
          //of getting stuck in local optimums
          var magnitude = r.NextDouble();
          var positive = true;
          var iterCount = 0f;
          var magnitudeChange = 5;
          while (iterCount &lt; maxIters)
          {
              iterCount++;
              if (positive)
              {   //Try adding a positive value to the property
                  property += magnitude;
                  //Evaluate the fitness
                  var fitness = fitnessFunction(network);
                  if (fitness == lastFitness)
                  {   //No change in fitness, increase the magnitude and re-try
                      magnitude *= magnitudeChange;
                      property = lastValue;
                  }
                  else if (fitness &lt; lastFitness)
                  {   //This change decreased the fitness (bad)
                      //Put the property back and try going in the negative direction
                      property = lastValue;
                      positive = false;
                  }
                  else
                  {   //This change increased the fitness (good)
                      //on the next iteration we will try 
                      //to apply the same change again
                      lastFitness = fitness;
                      lastValue = property;
                      //don't increase the iteration count as much
                      //if a good change was made
                      iterCount -= 0.9f;
                  }
              }
              else
              {   //Try adding a negative value to the property
                  property -= magnitude;
                  var fitness = fitnessFunction(network);
                  if (fitness == lastFitness)
                  {
                      //No change in fitness, increase the magnitude and re-try
                      magnitude *= magnitudeChange;
                      property = lastValue;
                  }
                  else if (fitness &lt; lastFitness)
                  {
                      //This change decreased the fitness (bad)
                      //Now we know that going in the positive direction 
                      //and the negative direction decreases the fitness
                      //so make the magnitude smaller as we are probably close to an optimum
                      property = lastValue;
                      magnitude /= magnitudeChange;
                      positive = true;
                  }
                  else
                  {
                      //This change increased the fitness (good)
                      //Continue in same direction
                      lastFitness = fitness;
                      lastValue = property;
                      iterCount -= 0.9f;
                  }
              }
              //Check bounds to prevent math functions overflowing
              if (property &gt; 100)
              {
                  property = 100;
                  lastFitness = fitnessFunction(network);
                  return lastFitness;
              }
              else if (property &lt; -100)
              {
                  property = -100;
                  lastFitness = fitnessFunction(network);
                  return lastFitness;
              }
          }
          return lastFitness;
      }
  </code></pre>
  
  <p>The fitness function is very expensive, so it should be called as little as possible. I'm looking for any improvements in getting to the local optimum with less calls to the fitness function. 
  Getting stuck in a local optimum is not too much of a concern, I have graphed the fitness function against the value of different weights and biases in the network, and it looks like usually there are between 1-3 local optimums in the graph. If the network remains at the same fitness for a few passes then I can add a parameter to this function to attempt restarting the hill climbing from a random value.</p>