Apply Labeled LDA on large data
You matrix will be sparse, so the memory usage will be inefficient. Try to look on LSH - locality sensitive hashing for clustering.

https://en.wikipedia.org/wiki/Locality-sensitive_hashing