anchor.diff:      </div>
   </div>
   
  -A conventional choice for the activation function had been the [sigmoid](http://en.wikipedia.org/wiki/Sigmoid_function) $\sigma (z) = 1 / (1 + e^{-z})$ for a long period because of its simple derivative and gain control properties. Recently the hyperbolic tangent $\tanh$ and the [rectified linear unit](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) $\textrm{ReLU}(z) = \max(0, z)$ have been more popular choices due to their convergence and training performance characteristics.
  -
  -Now let's write the network evaluation code and a function for creating a given network configuration and initializing the weights and biases with small random values. In practice, proper weight initialization would depend on the network structure and the type of activation functions used, as it has been demonstrated to have an important effect on training convergence.
  +A conventional choice for the activation function had been the [sigmoid](http://en.wikipedia.org/wiki/Sigmoid_function) $\sigma (z) = 1 / (1 + e^{-z})$ for a long period because of its simple derivative and gain control properties. Recently the hyperbolic tangent $\tanh$ and the [rectified linear unit](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) $\textrm{ReLU}(z) = \max(0, z)$ have been more popular choices due to their convergence and performance characteristics.
  +
  +Now let's write the network evaluation code and a function for creating a given network configuration and initializing the weights and biases with small random values. In practice, proper weight initialization has been demonstrated to have an important effect on training convergence and it would depend on the network structure and the type of activation functions used.
   
   Network evaluation is implemented using linear algebra, where we have a weight matrix $\mathbf{W}^l$ holding the weights of all neurons in layer $l$. The elements of this matrix $w_{ij}$ represent the weight between the $j$-th neuron in layer $l - 1$ and the $i$-th neuron in layer $l$. We also have a bias vector $\mathbf{b}^l$ holding the biases of all neurons in layer $l$ (one bias per neuron). Thus evaluating the network is just a matter of computing the activation
   
   $$$
     \mathbf{a}^l = \mathbf{W}^l \mathbf{a}^{l - 1} + \mathbf{b}^l \; ,
   
  -for each layer and passing the activation vector (output) of each layer as the input to the next layer, until we have the network output as the output vector of the last layer.
  +for each layer and passing the activation vector (output) of each layer as the input to the next layer, until we get the network output as the output vector of the last layer.
   *)
   
   let runLayer (x:DV) (l:Layer) =
  
positive.code: 
  // Initialize a fully connected feedforward neural network
  // Weights and biases between -0.5 and 0.5
  // l : number of inputs and number of neurons in each subsequent layer
  let createNetwork (l:int[]) =
      {layers = Array.init (l.Length - 1) (fun i ->
          {W = DM.init l.[i + 1] l.[i] (fun _ _ -> -0.5 + rnd.NextDouble())
  
negative.code: Neural Networks
  ===============
  
  [Artificial neural networks](http://en.wikipedia.org/wiki/Artificial_neural_network) are computational models inspired by biological nervous systems, capable of approximating functions that depend on a large number of inputs. A network is defined by a connectivity structure and a set of weights between interconnected processing units ("neurons"). Neural networks "learn" a given task by tuning the set of weights under an optimization procedure.
  
  Let's create a [feedforward neural network](http://en.wikipedia.org/wiki/Feedforward_neural_network) with DiffSharp and implement the [backpropagation](http://en.wikipedia.org/wiki/Backpropagation) algorithm for training it. As mentioned before, backpropagation is just a special case of reverse mode AD.
  
  We start by defining our neural network structure.
  
  