What do mean and variance mean for high dimensional data?
It's just the expansion of one dimensional mean and standard deviation. Suppose that you are trying to estimate the weight of a person and you have two inputs, salary and height. For finding the weight, you have two inputs which are of different scales, so you try to find the mean and variance of each feature, salary and height, separately using the data samples. Suppose you have two data samples in a tuple like (salary, height). They are as follows:


  (10000, 180)
  (5000, 175)


Although the first guy may seem so rich, the real point about the first and second data samples is that their scale is not the same. The approach is like this:


Find the mean of first feature among data samples, salary here. (10000 + 5000) / 2 = 7500
Find the mean of second feature among data samples, height here. (180 + 175) / 2 = 177.5
Find the standard deviation of first feature, among data samples, which is 2500 for salary.
Find the standard deviation of second feature, among data samples, which is 2.5 for height. 
Reduce the amount of mean from corresponding feature for each data sample and divide each with the standard deviation of corresponding feature.
(10000 - 7500) / 2500 = 1
(5000 - 7500) / 2500 = -1

(180 - 177.5) / 2.5 = 1
(175 - 177.5) / 2.5 = -1


Data samples which have been normalized now are like as follows:


  (1, 1)
  (-1, -1)


Whenever you normalize your data, your cost function would be so easier to learn. The weights don't have to struggle to reach to high values in situations where your features are not in same range.