title: Issue running Cosine Similarity on 50k records
body: <p>I am using spark 1.6 cosine similarity algorithm in spark mllib.</p>
  
  <p>Input: 50k documents' text with ids in a dataframe.</p>
  
  <p>Processing :</p>
  
  <ul>
  <li>Tokenized the texts</li>
  <li>Removed stop words</li>
  <li>Generated vectors(size=300) using word2Vec</li>
  <li>Generated RowMatrix</li>
  <li>Transposed it</li>
  <li>Used columnSimilarities method with threshold 0.1.(Also tired higher 
  values)</li>
  </ul>
  
  <p>Output is nxn matrix.</p>
  
  <p>I am using this spark submit</p>
  
  <p>spark-submit --master yarn --conf "spark.kryoserializer.buffer.max=256m" --num-executors 60 --driver-memory 10G --executor-memory 15G --executor-cores 5 --conf "spark.shuffle.service.enabled=true" --conf "spark.yarn.executor.memoryOverhead=2048" noname.jar xyzclass </p>
  
  <p>I am also doing 400 partitions. 
  But I am getting out of memory issues. I have tired different combinations for partitions and number of executors but failed to run it successfully. However I am able to run it successfully for 7k records with vector size 50 in less than 7mins. Any suggestions how I can make it run on 50K records?</p>