anchor.diff:  Neural Networks
   ===============
   
  -[Artificial neural networks](http://en.wikipedia.org/wiki/Artificial_neural_network) are computational models inspired by biological nervous systems, capable of approximating functions depending on a large number of inputs. A network is defined by a connectivity structure and a set of weights between interconnected processing units ("neurons"). Optimizing, or tuning, the set of weights for a given task makes neural networks capable of learning.
  -
  -Let's create a [feedforward neural network](http://en.wikipedia.org/wiki/Feedforward_neural_network) and use DiffSharp for implementing the [backpropagation](http://en.wikipedia.org/wiki/Backpropagation) algorithm for training it. As mentioned before, backpropagation is just a special case of reverse mode AD.
  +[Artificial neural networks](http://en.wikipedia.org/wiki/Artificial_neural_network) are computational models inspired by biological nervous systems, capable of approximating functions that depend on a large number of inputs. A network is defined by a connectivity structure and a set of weights between interconnected processing units ("neurons"). Neural networks "learn" a given task by tuning the set of weights under an optimization procedure.
  +
  +Let's create a [feedforward neural network](http://en.wikipedia.org/wiki/Feedforward_neural_network) with DiffSharp and implement the [backpropagation](http://en.wikipedia.org/wiki/Backpropagation) algorithm for training it. As mentioned before, backpropagation is just a special case of reverse mode AD.
   
   We start by defining our neural network structure.
   
  
positive.code:     static member Load(filename) = MNIST.Load(filename, System.Int32.MaxValue)
  
  (**
  Let's load 10,000 training images
  *)
  
  let mnistTrainX = MNIST.Load("C:/datasets/MNIST/train-images.idx3-ubyte", 10000)
  let mnistTrainY = MNIST.Load("C:/datasets/MNIST/train-labels.idx1-ubyte", 10000)
  
  (**
  Loss
  *)
  
  let cross (x:DM) (y:DM) =
      (x |> DM.toCols |> Seq.mapi (fun i v -> (logsumexp v) - v.[int (float y.[0, i])]) |> Seq.sum) / x.Cols
  
  (**
  Create a network
  *)
  
  let l0 = {W = DM.init 300 785 (fun _ _ -> -0.075 + 0.15 * rnd.NextDouble())
            b = DV.init 300 (fun _ -> 0.)
            a = tanh}
  
  let l1 = {W = DM.init 10 300 (fun _ _ -> -0.075 + 0.15 * rnd.NextDouble())
            b = DV.init 10 (fun _ -> 0.)
            a = id}
  
  let nn = {layers = [|l0; l1|]}
  
  (**
  Train
  *)
  
  //backprop' nn 0.1 0.005 1000 1000 cross mnistTrainX mnistTrainY
  
  (**
  Test
  *)
  
  let mnistTestX = MNIST.Load("C:/datasets/MNIST/t10k-images.idx3-ubyte", 5)
  let mnistTestY = MNIST.Load("C:/datasets/MNIST/t10k-labels.idx1-ubyte", 5)
  
  let testY = runNetwork' mnistTestX nn |> primal
  let testError = cross testY mnistTestY
  let testPredict = testY |> DM.toCols |> Seq.map DV.maxIndex |> Seq.toArray
  
  for i = 0 to testY.Cols - 1 do
      printfn "Predicted label: %i" testPredict.[i]
  
negative.code:     Array.fold runLayer' x n.layers
  
  (**
  The backpropagation code given previously computed the loss over the whole set of training cases at each iteration and used simple gradient descent to iteratively decrease this loss. When the training set is large, this leads to training time bottlenecks. 
  
  In practice, backpropagation is combined with stochastic gradient descent (SGD), which makes the duration of each training iteration independent from the training set size (also see the [SGD example](examples-stochasticgradientdescent.html)). Furthermore, instead of using one random case at a time to compute the loss, SGD is used with "minibatches" of more than one case (a small number compared to the full training set size). Minibatches allow us to exploit the matrix-matrix multiplication trick for performance and also have the added benefit of smoothing the SGD estimation of the true gradient.
  
  