Deepmind just published a paper about a differentiable neural computer, which basically combines a neural network with a memory. The idea is to teach the neural network to create and recall useful explicit memories for a certain task. This complements the abilities of a neural network well, because NNs only store knowledge implicitly in the weights and the information used to work on a single task is only stored in the activation of the network and degrades quickly the more information you add. (LSTMs are one try to slow down this degradation of short term memories, but it still happens.)Now, instead of keeping the necessary information in the activation, they presumably keep the addresses of memory slots for specific information in the activation, so these should also be subject to degradation. My question is why this approach should scale. Shouldn't a somewhat higher number of task specific information once again overwhelm the networks capability of keeping the addresses of all the appropriate memory slots in its activation?
How would Deepmind's new differentiable neural computer scale?
deep learning;ai design