What it means for a neural network to memorize data
Memorization is the same as overfitting.  The memory is implicitly represented by your weights. If your network does have enough parameters it will be able to memorize/overfit.

In order to understand what I mean by overfitting and memorization let us look at the polynomial regression

$$y_n=w_0+w_1x_n+w_2x_n^2+ \varepsilon.$$

We have three coefficients. If we only had three data points (which do not perfectly lie on a line) we could fit a quadratic regression equation without any error. Hence, the model would memorize the data by using three coefficients. 
We would have a loss of zero, but as you know this result would also be very likely overfitting the model to the data.