anchor.diff:  
   == Conclusion
   
  -We saw pretty good improvements by making a relatively simple change to the neuron computation model. Experiments used `ReLU` which effectively attenuates negative values. This limits the neuron to only communicate information about the degree of similarity in matching with the weight template. ReLU effectively squishes _dissimilarity_ or the _inverse_ of weight template information to zero indicating no similarity. Perhaps it is useful if a neuron can communicate dissimilarity information instead. For example, lack of a specific stripe pattern might increase the probability of one cat species over another. 
  +We saw pretty good improvements by making a relatively simple change to the neuron computation model. Experiments used `ReLU` which effectively attenuates negative values. This limits the neuron to only communicate information when the angle between \(X\) and \(W\) lies between \([-\frac{\pi}{2}, \frac{\pi}{2}]\). Perhaps it is useful if a neuron could also communicate the _lack of_ similarity or the _inverse_ of weight template information. For example, the lack of a specific stripe pattern might increase the networks confidence that the output is more likely to be one cat species over another. 
   
  -First thought might be to use a `linear` activation function, but this limits the representational power. A quick experiment with `ELU` activation, however, did not show any significant improvement over ReLU. +One way to remedy this problem might be to use an activation function that allows negative values. A quick experiment with `ELU` activation, however, did not show any significant improvement over `ReLU`.
positive.code: 
  * In a complex neural network, every neuron is matching the input against some template encoded via weights and forwarding that result to the subsequent layers. In case of image recognition, the first layer neurons might match whether the input contains vertical edge, horizontal edge and so on. The circle detection neuron might use various edge matching computation results from previous layers to detect the shape patterns.
  
  So far so good. What is the purpose of activation function? To understand the motivation, consider the geometric perspective of the computation in terms of linear algebra.
  
  \( W.X = \begin{bmatrix} w_{1} \\ w_{2} \\ \vdots \\ w_{n} \end{bmatrix} \begin{bmatrix} x_{1} & x_{2} & \cdots & x_{n} \end{bmatrix} \)
  
  Geometrically, this can be interpreted as a linear transformation of \(X\). Additionally, there is a bias term to shift the origin. So, the computation \(W.X + b\) can be interpreted as an affine transformation. Without an activation function, multiple linear transformations within each layer \(W_{L}\) can be condensed into a single layer with \(W = \prod_{L}W_{L}\). A single layered linear neural network is pretty limited.
  
  Lets put all this togther to interpret what a multi-layered neural network might be doingfootnote:[This is my own interpretation and might as well be incorrect.]:
  
  * Lets say we input a \(100 \times 100\) image to a vggnet. The input tensor \(X\) can be thought of as a vector with 10000 basis. Each basis is representing some positional component in an image.
  * We know that the very first conv layer learns gabor filter like edge detection weight templates. Lets assume that a single conv-filter generates \(100 \times 100\) output (with the appropriate padding). This can be seen as _transforming_ input from positional basis vector space to edge-like basis vector space. Insead of measuring the pixel intensity, the new vector space measures the similarity of the positional input patch with respect to a template (\(3 \times 3\) filter weights). In this manner, every layer _transforms_ the data into a more meaningful basis vector space, eventually leading to basis vectors consisting of output categories.
  
  == Motivation
  
  
negative.code: = Exploring alternative neural computational models.
  :hp-tags: deep learning
  
  Despite different variations of neural networks architectures, the computational model of a neuron hasn't changed since inception. Every neuron has \(n\) incomming inputs \(X = (x_{1}, x_{2}, ..., x_{n})\) with weights \(W = (w_{1}, w_{2}, ..., w_{n})\). A neuron then computes \(W.X = \sum_{i=0}^{n}w_{i}.x_{i}\), forwards it through some nonlinearity like sigmoid or ReLU to generate the outputfootnote:[Technically, bias is involved, but i am excluding it to keep the discussion focused.].
  
  [.text-center]
  .Computation model of a neuron
  image::sim_neuron/neuron_model.jpeg[neuron_model, 400]
  {empty} +
  
  As far as i am aware, all architectures build on this fundamental computational model. Let us examine what the computation is conceptually doing in a step by step fashion.
  
  * Each input component \(x_{i}\) is being magnified or diminished when multiplied with a scaling factor \(w_{i}\). This can be interpreted as enhancing or diminishing the contribution of some pattern component \(x_{i}\) to the output decision. Think of these weights as _knobs_ that range from \([-\infty, +\infty]\). The task of learning is to _tune_ these knobs to the correct value.
  
  