title: Mahout RowSimilarityJob on big document corpus
body: <p>I am trying to use Mahout to compute row similarities in a matrix containing 2 million rows. This matrix is produced taking the output of mahout ssvd with rank 400 and then using mahout rowid to transform it in the appropriate format for RowSimilarityJob.</p>
  
  <p>When the job reaches the CooccurrencesMapper it starts emitting a vector for each couple of non-zero elements in a column. After a couple of hours the job starts failing. </p>
  
  <p>I tried to use the --maxObservationsPerColumn parameter, but if i set it to a value too low the results are incorrect, if i set it to a value higher than 30000 it fails. </p>
  
  <p>The problem is that for each map input it emits ~10GB of values.</p>
  
  <p>I invoke the job like this:</p>
  
  <blockquote>
    <p>mahout/bin/mahout rowsimilarity -Dmapred.reduce.child.java.opts=-Xmx4G
    -Dmapred.map.child.java.opts=-Xmx4G -Dmapred.reduce.tasks=20 -Dio.sort.mb=1000 -Dio.sort.factor=3 --tempDir test_row_similarity --input "UHalfSigma_matrix_400/matrix" --output "doc_lsa_similarity_test/" --numberOfColumns 400
    --maxSimilaritiesPerRow 35 --maxObservationsPerColumn 50000 -tr 0.4 --similarityClassname SIMILARITY_COSINE -ow --excludeSelfSimilarity true</p>
  </blockquote>
  
  <p>I use a cluster of 3 nodes:</p>
  
  <pre><code>- cpu 12 core, RAM 64 GB, HDD 256GB
  - cpu 12 core, RAM 128 GB, HDD 2 TB
  - cp 8 core, RAM 24 GB, HDD 4 TB
  </code></pre>
  
  <p>Thank you for your help</p>