anchor.diff:  Neural Networks
   ===============
   
  -[Artificial neural networks](http://en.wikipedia.org/wiki/Artificial_neural_network) are computational models inspired by biological nervous systems, capable of approximating functions depending on a large number of inputs. A network is defined by a connectivity structure and a set of weights between interconnected processing units ("neurons"). Optimizing, or tuning, the set of weights for a given task makes neural networks capable of learning.
  -
  -Let's create a [feedforward neural network](http://en.wikipedia.org/wiki/Feedforward_neural_network) and use DiffSharp for implementing the [backpropagation](http://en.wikipedia.org/wiki/Backpropagation) algorithm for training it. As mentioned before, backpropagation is just a special case of reverse mode AD.
  +[Artificial neural networks](http://en.wikipedia.org/wiki/Artificial_neural_network) are computational models inspired by biological nervous systems, capable of approximating functions that depend on a large number of inputs. A network is defined by a connectivity structure and a set of weights between interconnected processing units ("neurons"). Neural networks "learn" a given task by tuning the set of weights under an optimization procedure.
  +
  +Let's create a [feedforward neural network](http://en.wikipedia.org/wiki/Feedforward_neural_network) with DiffSharp and implement the [backpropagation](http://en.wikipedia.org/wiki/Backpropagation) algorithm for training it. As mentioned before, backpropagation is just a special case of reverse mode AD.
   
   We start by defining our neural network structure.
   
  
positive.code:                 l.W <- l.W |> makeReverse i
                  l.b <- l.b |> makeReverse i
  
              let error:D = loss (runNetwork' mbX n) mbY
              error |> reverseProp (D 1.)
  
              for l in n.layers do
                  l.W <- primal (l.W.P - eta * l.W.A)
                  l.b <- primal (l.b.P - eta * l.b.A)
  
              printfn "Epoch %i, minibatch %i, error %f" j b (float error)
              b <- b + 1
              if float error >= epsilon then j <- timeout
          j <- j + 1
  
  (**
  The Obligatory MNIST Example
  ----------------------------
  
  The MNIST (for "Mixed National Institute of Standards and Technology") database of handwritten digits is
  
  
  The following code reads the standard MNIST files into matrices.
  *)
  
negative.code: // The backpropagation algorithm
  // n: network to be trained
  // eta: learning rate
  // epochs: number of training epochs
  // x: training input vectors
  // y: training target vectors
  let backprop (n:Network) eta epochs (x:DV[]) (y:DV[]) =
      let i = DiffSharp.Util.GlobalTagger.Next
      seq {for j in 0 .. epochs do
              for l in n.layers do
                  l.W <- l.W |> makeReverse i
                  l.b <- l.b |> makeReverse i
  
              let L = Array.map2 (fun x y -> DV.l2normSq (y - runNetwork x n)) x y |> Array.sum
              L |> reverseProp (D 1.) // Propagate adjoint value 1 backward
  
              for l in n.layers do
                  l.W <- primal (l.W.P - eta * l.W.A)
                  l.b <- primal (l.b.P - eta * l.b.A)
  
              printfn "Iteration %i, loss %f" j (float L)
              yield float L}
  
  (**
  
  Using reverse mode AD here has two big advantages: (1) it makes the backpropagation code succinct and straightforward to write and maintain; and (2) it allows us to freely code our neural network without the burden of coding derivatives or modifying the backpropagation code accordingly.
  
  We can now test the algorithm by training some networks.
  
  