title: Applying a function between specific pairs of columns in a matrix in R
body: <p>I am generating a matrix using the lsa package in R. After the matrix is created, I would like to calculate the cosine similarity between <em>specific</em> pairs of documents (columns) in the matrix.</p>
  
  <p>Currently, I am doing this with nested for-loops, and it is <em>monstrously</em> slow. In the code below, there are 150 <em>sourceIDs</em> and 6413 <em>targetIDs</em>, for a total of 961.950 comparisons. After an hour and a half on my number-crunching machine, it has only gotten through ~300k of them.</p>
  
  <p>For more info, <em>sourceIDs</em> and <em>targetIDs</em> are vectors of column names, loaded in from two files containing those names. I want to apply the cosine function between all of the source->target pairs. The columns are indexed by the document name, which is a string.</p>
  
  <p>I am sure there is a much faster way of doing this with <em>apply</em>, but I just cant wrap my head around it.</p>
  
  <pre><code>library(lsa)
  
  # tf function
  real_tf &lt;- function(m)
  {
      return (sweep(m, MARGIN=2, apply(m, 2, max), "/"))
  }
  
  #idf function
  real_idf &lt;- function(m)
  {
      df = rowSums(lw_bintf(m), na.rm=TRUE)
      return (log(ncol(m)/df))
  }
  
  #load corpus
  lsa.documents &lt;- textmatrix(args[1], minWordLength=1, minDocFreq=0)
  
  # compute tf-idf
  lsa.weighted_documents &lt;- real_tf(lsa.documents) * real_idf(lsa.documents)
  
  # compute svd
  lsa.nspace &lt;- lsa(lsa.weighted_documents, dims = as.integer(args[5]))
  lsa.matrix &lt;- diag(lsa.nspace$sk) %*% t(lsa.nspace$dk)
  
  # compute similarities
  lsa.sourceIDs &lt;- scan(args[2], what = character())
  lsa.targetIDs &lt;- scan(args[3], what = character())
  lsa.similarities &lt;- data.frame(SourceID=character(), TargetID=character(), Score=numeric(), stringsAsFactors=FALSE)
  k &lt;- 1
  for (i in lsa.sourceIDs)
  {
      for (j in lsa.targetIDs)
      {
          lsa.similarities[k,] &lt;- c(i, j, cosine(lsa.matrix[,i], lsa.matrix[,j]))
          k &lt;- k + 1
      }
  }
  lsa.ranklist &lt;- lsa.similarities[order(lsa.similarities$Score, decreasing=TRUE),]
  
  # save ranklist
  write.table(lsa.ranklist, args[4], sep="\t", quote=FALSE, col.names=FALSE, row.names=FALSE)
  </code></pre>
  
  <p>Edit: Reproducible example</p>
  
  <pre><code># cosine function from lsa package
  cosine &lt;- function( x, y )
  {
      return ( crossprod(x,y) / sqrt( crossprod(x)*crossprod(y) ) )
  }
  
  theMatrix &lt;- structure(c(-0.0264639232505822, -0.0141165039351167, -0.0280459775632757, 
  -0.041211247161448, -0.00331565717239375, -0.0291161345945683, 
  -0.0451167802746869, -0.0116214407383401, -0.0381080747718958, 
  -1.36693644389599, 0.274747343110076, 0.128100677705483, -0.401760905661056, 
  -1.24876927957167, 0.368479552862631, -0.459711112157286, -0.544344448332346, 
  -0.765378939625159, -1.28612431910459, 0.293455499695499, 0.025167452173962
  ), .Dim = c(3L, 7L), .Dimnames = list(NULL, c("doc1", "doc2", "doc3", 
  "doc4", "doc5", "doc6", "doc7")))
  
  sources &lt;- c("doc1", "doc2", "doc3")
  targets &lt;- c("doc4", "doc5", "doc6", "doc7")
  
  similarities &lt;- data.frame(SourceID=character(), TargetID=character(), Score=numeric(), stringsAsFactors=FALSE)
  k &lt;- 1
  
  for (i in sources)
  {
      for (j in targets)
      {
          similarities[k,] &lt;- c(i, j, cosine(theMatrix[,i], theMatrix[,j]))
          k &lt;- k + 1
      }
  }
  
  ranklist &lt;- similarities[order(similarities$Score, decreasing=TRUE),]
  write.table(ranklist, "C:\\Temp\\outputfile.txt", sep="\t", quote=FALSE, col.names=FALSE, row.names=FALSE)
  </code></pre>
  
  <p>Which produces (outputfile.txt):</p>
  
  <pre><code>doc1    doc6    0.962195242094352
  doc3    doc6    0.893461576046585
  doc2    doc6    0.813856201398669
  doc2    doc7    0.768837903803964
  doc2    doc4    0.730093288388069
  doc3    doc7    0.675640649189972
  doc3    doc4    0.635982900340315
  doc1    doc7    0.53871688669971
  doc1    doc4    0.499235059782688
  doc1    doc5    0.320383772495164
  doc3    doc5    0.226751624753921
  doc2    doc5    0.144680489733846
  </code></pre>