How is mean square displacement defined in Brownian motion?
The mean squared distance is a measure of deviation: how far away two things are. 

Let's say we want to find the MSD of a particle from it's origin. What we do is find the square distance at specific time intervals, say every second. We then make a table of data:

$$\begin{array}{c|c|} 
 \text{time} &amp; \text{square distance} \\ \hline
 1 &amp; 3.6 \\ \hline
  2 &amp; 3.8 \\ \hline
 3 &amp; 3.4 \\ \hline
 4 &amp; 3.2 \\ \hline
\end{array}$$

The formula for the average is adding everything up and dividing by the number of things, so in this example:

$$MSD = \frac{3.6+3.8+3.4+3.2}{4}$$

What I've done above is stated more generally, and much more elegantly, as the formula you stated,

$$MSD = 1/T * \sum_\text{t=1}^\text{T} (x(t)-x_\text{0})^2$$

Your cumulative sum formula doesn't make intuitive sense, it instead finds the sum of the mean square distance at every time increment (there is no reason why you'd want to do that)