anchor.diff:  
   Geometrically, this can be interpreted as a linear transformation of \(X\). Additionally, there is a bias term to shift the origin. So, the computation \(W.X + b\) can be interpreted as an affine transformation. Without an activation function, multiple linear transformations within each layer \(W_{L}\) can be condensed into a single layer with \(W = \prod_{L}W_{L}\). A single layered linear neural network is pretty limited.
   
  -Lets put all this togther to interpret what a multi-layered neural network might be doing: footnote:[This is my own interpretation and might as well be incorrect.]
  +Lets put all this togther to interpret what a multi-layered neural network might be doingfootnote:[This is my own interpretation and might as well be incorrect.]:
   * Lets say we input a \(100 \times 100\) image to a vggnet. The input tensor \(X\) can be thought of as a vector with 10000 basis. Each basis is representing some positional component in an image.
   * We know that the very first conv layer learns gabor filter like edge detection weight templates. Lets assume that a single conv-filter generates \(100 \times 100\) output (with the appropriate padding). This can be seen as _transforming_ input from positional basis vector space to edge-like basis vector space. Insead of measuring the pixel intensity, the new vector space measures the similarity of the positional input patch with respect to a template (\(3 \times 3\) filter weights). In this manner, every layer _transforms_ the data into a more meaningful basis vector space, eventually leading to basis vectors consisting of output categories.
   
  
positive.code: 
  If we view a neuron as fundamental template matching unit, then various magnitude (length) of \(X\) and \(W\) should yield the same output. If we constrain \(W\) to be a unit vector (\(|W| = 1\)), then backprop would only need to search along points on a hypersphere of unit radius. Although the hypersphere contains infinitely many points, with sufficient discretization, the search space intuitively _feels_ a lot smaller than the entire weight vector space.
  
  The proposal is simple. Each neuron should compute: \(Y = \frac{W.X}{|W||X|}\). From a layer perspective, instead of computing \(W \times X\) we are now computing \(\frac{W \times X}{|X|}\).
negative.code: 
  * Each input component \(x_{i}\) is being magnified or diminished when multiplied with a scaling factor \(w_{i}\). This can be interpreted as enhancing or diminishing the contribution of some pattern component \(x_{i}\) to the output decision. Think of these weights as _knobs_ that range from \([-\inf, +\inf]\). The task of learning is to _tune_ these knobs to the correct value.
  
  * The computation \(W.X\) is then enhancing/diminishing various input components of vector \(X\). In terms of linear algebra, the dot product of \(W\) and \(X\) can be interpeted as computing similarity. Consider \(X = (x_{1}, x_{2})\) and \(W = (w_{1}, w_{2})\). If you imagine plotting these vectors on a 2D graph paper, \(\frac{W.X}{|W||X|}\) is the cosine of angle between these two vectors. When these two vectors align (point in the same direction), the value of \(W.X\) is maximized indicating high degree of similarity. From this point of view, neuron is fundametally a pattern matching unit which outputs a large value when it matches a specific pattern (encoded by weights) in the inputfootnote:[The correct weight vectors are learned using backpropogation.].
  
  * In a complex neural network, every neuron is matching the input against some template encoded via weights and forwarding that result to the subsequent layers. In case of image recognition, the first layer neurons might match whether the input contains vertical edge, horizontal edge and so on. The circle detection neuron might use various edge matching computation results from previous layers to detect the shape patterns.
  
  