What it means for a neural network to memorize data I've seen at many places that sometimes neural networks simply memorizes training data. What that means actually ?

Neural network consists of bunch of weights which gets trained and outputs based on input data. It'll output different thing for different input. Where does this memorization comes into play ?
Memorization is the same as overfitting.  The memory is implicitly represented by your weights. If your network does have enough parameters it will be able to memorize/overfit.

In order to understand what I mean by overfitting and memorization let us look at the polynomial regression

$$y_n=w_0+w_1x_n+w_2x_n^2+ \varepsilon.$$

We have three coefficients. If we only had three data points (which do not perfectly lie on a line) we could fit a quadratic regression equation without any error. Hence, the model would memorize the data by using three coefficients. 
We would have a loss of zero, but as you know this result would also be very likely overfitting the model to the data.