What does it mean by “t-SNE retains the structure of the data”?
You should break this down one step further: retaining local structure and retaining global structure.




Other well-understood methods, such as Principal Component Analysis are great at retaining global structure, because it looks at ways in which a dataset's variance is retained, globally, across the entire dataset.
t-SNE works differently, by looking at locally appearing datapoints. It does this by computing a metric between each datapoint and a given number of neighbours - modelling them as being within a t-distributed distribution (hence the name: t-distributed Stochastic Neighbourhood Embedding). It then tries to find an embedding, such that neighbours in the original n-dimensional space, are also found close together in the reduced (embedded) dimensional space. It does this by minimising the KL-divergence between the before and after datapoint distributions, $\mathbb{P}$ and $\mathbb{Q}$ respectively.


This method has the benefit of retaining local structure - so clusters in the low dimensional space should be interpretable as datapoints that were also very similar in the high dimensional space. t-SNE works remarkedly well on many problem, however there are a few things to watch out for:


Because we know have some useful local structure retained, we essentially trade that off for ability in retaining global structure. This equates to you not being able to really compare e.g. 3 clusters in the final embedding, where 2 are close together and 1 is far away. This does not mean they were also far away from each other in the original space. 
t-SNE can be very sensitive to its perplexity parameter. In fact, you might get different results with the three-cluster example in point 1, using an only slightly different perplexity value. This value can indeed be roughly equated to "how many points shall we inlude in the t-distribution to find neighbours of a datapoint" - it essentially gives the area which is encompassed in the t-distribution.




I would recommend watching this lecture by the author of t-SNE, Laurens van der Maaten, as well as getting some intuition for t-SNE and it's parameters using this great visual explanation.

There are also some good answers here on CrossValidated with a little more technical information.