Using t-SNE to track progress of a word vector embedding model. Pitfalls? I've been training a word2vec/doc2vec model on a large amount of text. I recently stumbled across the t-SNE package, and am finding it wonderful at finding hidden structure in high-dimensional data.

Can t-SNE be used as a way of tracking the progress of a hard machine learning task like this - where the model's understanding goes from unintelligible nonsense to something with hidden structure?

I have seen examples of the MNIST data set on t-SNE where all the individual numbers cluster well with each other. (as explained in this answer)


As I increase the number of vectors in the doc2vec model and the size of the training set, I start to see clumping (if you squint) in the t-SNE plot. So far, these clumps are mainly associated with posts of very similar wording (one clump is mainly "Good morning/evening!" tweets). (Picture was generated with perplexity of 400)



How much additional clumping can I expect to see as the model is improved? Is this indicative that the model is, in fact, improving and learning deeper connections between words/phrases? Or have these t-SNE plots settled into the form they'll always take?

EDIT: I have since realised that the apparent lack of clumping could be due to the data itself. MNIST separates out cleanly because there are generally no weird glyphs that look like halfway mutations between numbers. My dataset (twitter sentiment, 1.6 million tweets) is, for a lack of a better word, filled with unclassifiable drivel, and it seems entirely probable that the homogeneous forest of points in the centre of the plot represents these sorts of tweets.
T-SNE is extremely useful for visualizing high-dimensional data in lower-dimensional space. However, t-SNE can have several gotchas, including comparing cluster sizes. The t-sne algorithm tries to even out cluster sizes by expanding dense clusters and contracting sparse clusters. Thus, it is not straightforward to directly compare clusters across different runs.

"How to Use t-SNE Effectively"  goes into greater detail about common pitfalls of using the technique.