A complete model describes the mathematics of how the future state of activity of each neuron depends on the known present or previous activity of all the neurons. In the original Hopfield model of associative memory, the variables were binary, and the dynamics were described by a one-at-a-time update of the state of the neurons. An energy function quadratic in the was defined, and the dynamics consisted of changing the activity of each single neuron only if doing so would lower the total energy of the system. This same idea was extended to the case of being a continuous variable representing the output of neuron , and being a monotonic function of an input current. The dynamics became expressed as a set of first-order differential equations for which the "energy" of the system always decreased. The energy in the continuous case has one term which is quadratic in the (as in the binary model), and a second term which depends on the gain function (neuron's activation function). While having many desirable properties of associative memory, both of these classical systems suffer from a small memory storage capacity, which scales linearly with the number of input features. Dense Associative Memories (also known as the modern Hopfield networks) are generalizations of the classical Hopfield Networks that break the linear scaling relationship between the number of input features and the number of stored memories. This is achieved by introducing stronger non-linearities (either in the energy function or neurons’ activation functions) leading to super-linear (even an exponential) memory storage capacity as a function of the number of feature neurons. The network still requires a sufficient number of hidden neurons. The key theoretical idea behind the modern Hopfield networks is to use an energy function and an update rule that is more sharply peaked around the stored memories in the space of neuron’s configurations compared to the classical Hopfield Network. Discrete variables A simple example of the modern Hopfield network can be written in terms of binary variables that represent the active and inactive state of the model neuron .In this formula the weights represent the matrix of memory vectors (index enumerates different memories, and index enumerates the content of each memory corresponding to the -th feature neuron), and the function is a rapidly growing non-linear function. The update rule for individual neurons (in the asynchronous case) can be written in the following form which