title: Memory initialization and addressing in Neural Turing Machines
body: <p>It is not clear to me how the NTM initializes its memory to support content-based addressing with cosine distance. It's clear the initial memory cell values can not be zero but even if you initialize them to some non-zero value (say 1.0), all the cells will look the same and the memory addressing mechanism will produce a uniform distribution over the memory cells. The shift mechanism will just shift this uniform distribution, producing the same uniform distribution and the sharpening might sharpen some value, mainly due to noise. </p>
  
  <p>So I don't see how the memory they described can be used beyond writing the same thing all over the place all the time (if you discount some noise).</p>
  
  <p>Am I missing something and can somebody enlighten me about memory initialization?</p>