Reasoning behind performance improvement with hopfield networks
Will try to formulate my understanding of the ideas in this paper, mention my own concerns that I see are relevant to your question, and see if we can identify any confusions along the way that might clarify the issue
On eq(6) of the relevant blog post, they identify the weight matrix of a discrete, binary Hopfield network as
$$ \boldsymbol{W} = \sum_i^N x_i x_i^T $$
with N raw stored entries, that are retrieved by iterating the initial guess $\xi$ with the following update rule
$$ \xi_{t+1} = \text{sgn}( \boldsymbol{W} \xi_{t} - b ) $$
Now to the paper in question, the update rule for the generalization they propose for continuous states that would be used is (eq 22 in OP):
$$ \xi_{t+1} = \boldsymbol{X} \text{softmax}( \beta \boldsymbol{X}^T \xi_{t} ) $$
Where $\boldsymbol{X} = (x_0, x_1, \dots , x_N ) $
The first substantial difference I see is that, while on the case of binary entries, all the weights of the network are encoded in the matrix $\boldsymbol{W}$, hence the storage is constant regardless of how many actual patterns are stored in it. In contrast, in this continuous case generalized rule, the $\boldsymbol{X}$ matrix seems to grow linearly with the size of entries, in fact it keeps all the stored entries directly. By their update rule, it doesn't seem to be a way around storing and keeping around the entire entries, and the update rule seems to only find a &quot;best fit&quot; among the entries, using the scalar dot product of the attention mechanism. I still think I might be missing something important here