anchor.diff:  
   == Conclusion
   
  -We saw pretty good improvements by making a relatively simple change to the neuron computation model. Experiments used `ReLU` which effectively attenuates negative values. This limits the neuron to only communicate information about the degree of similarity in matching with the weight template. ReLU effectively squishes _dissimilarity_ or the _inverse_ of weight template information to zero indicating no similarity. Perhaps it is useful if a neuron can communicate dissimilarity information instead. For example, lack of a specific stripe pattern might increase the probability of one cat species over another. 
  +We saw pretty good improvements by making a relatively simple change to the neuron computation model. Experiments used `ReLU` which effectively attenuates negative values. This limits the neuron to only communicate information when the angle between \(X\) and \(W\) lies between \([-\frac{\pi}{2}, \frac{\pi}{2}]\). Perhaps it is useful if a neuron could also communicate the _lack of_ similarity or the _inverse_ of weight template information. For example, the lack of a specific stripe pattern might increase the networks confidence that the output is more likely to be one cat species over another. 
   
  -First thought might be to use a `linear` activation function, but this limits the representational power. A quick experiment with `ELU` activation, however, did not show any significant improvement over ReLU. +One way to remedy this problem might be to use an activation function that allows negative values. A quick experiment with `ELU` activation, however, did not show any significant improvement over `ReLU`.
positive.code: = Exploring alternative neural computational models.
  :hp-tags: deep learning
  
  Despite different variations of neural networks architectures such as LSTMs, ConvNets, endcoder-decoder architectures and so on, the computational model of a neuron hasn't changed since inception. Every neuron has \(n\) incomming inputs \(X = (x_{1}, x_{2}, ..., x_{n})\) with weights \(W = (w_{1}, w_{2}, ..., w_{n})\). A neuron then computes \(W.X = \sum_{i=0}^{n}w_{i}.x_{i}\) and then uses some nonlinearity like sigmoid or ReLU to generate the outputfootnote:[Technically, bias is involved, but i am excluding it to keep the discussion focused.].
  
  [.text-center]
  .Computation model of a neuron
  image::sim_neuron/neuron_model.jpeg[neuron_model, 400]
  {empty} +
  
  All architectures build on this fundamental computational model. Let us examine what the computation is conceptually doing in a step by step fashion.
  
  * Each input component \(x_{i}\) is being magnified or diminished when multiplied with a scaling factor \(w_{i}\). This can be interpreted as enhancing or diminishing the contribution of some pattern component \(x_{i}\) to the output decision. Think of these weights as _knobs_ that range from \([-\infty, +\infty]\). The task of learning is to _tune_ these knobs to the correct value.
  
  
negative.code: 
  == Experiments
  
  I explored option 3 as it was the simplest to implement. The architecture is a simplified version of vggnet comprising of \(3 \times 3\) convolutions with ReLU activation and max pooling. I used cifar10 dataset augmented with 10% random shifts along image rows/cols along with a 50% chance of horizontal flip. `random_seed = 1337` was used to get consistent and reproducable results across trials.
  
  The model has 1,250,858 parameters and trained for 50 epochs with a batch size of 32 using categorical crossentropy loss with Adam optimizer.
  
  \(W_{norm}\) is calculated as:
  [source,python]
  