Purpose of visualizing high dimensional data? There are many techniques for visualizing high dimension datasets, such as T-SNE, isomap, PCA, supervised PCA, etc. And we go through the motions of projecting the data down to a 2D or 3D space, so we have a "pretty pictures". Some of these embedding (manifold learning) methods are described here.



But is this "pretty picture" actually meaningful? What possible insights can someone grab by trying to visualize this embedded space?

I ask because the projection down to this embedded space is usually meaningless. For example, if you project your data down to principal components generated by PCA, those principal components (eiganvectors) don't correspond to features in the dataset; they're their own feature space. 

Similarly, t-SNE projects your data down to a space, where items are near each other if they minimize some KL divergence. This isn't the original feature space anymore. (Correct me if I'm wrong, but I don't even think there is a large effort by the ML community to use t-SNE to aid classification; that's a different problem than data visualization though.)

I'm just very largely confused why people make such a big deal about some of these visualizations.
First of all your explanation about the methods are right. The point is that Embedding algorithms are not to only visualize but basically reducing the dimentionality to cope with two main problems in Statistical Data Analysis, namely Curse of Dimentionaliy and Low-Sample Size Problem so that they are not supposed to depict physically understood features and they are not only meaningful but also necessary for data analysis!

Actually the visualization is almost the last usage of embedding methods. Projecting high-dimensional data into a lower-dimension space helps to preserve the actual pair-wise distances (mainly Euclidean one) which get distorted in the high dimensions or capturing the most information embedded in the variance of different features.