R Function to Generate Predictions from Ratings and Save Results I am trying to generate predictions using the function below in combination with sapply. My actual datasets are very large, I am attempting to make 1.5 million predictions with this function and it is currently taking about 10 seconds per prediction which is... prohibitive to say the least.

mean_rating &lt;- function(df){
  #mean_rating &lt;- function(query_user,query_movie){
  user&lt;-df$user
  movie&lt;-df$movie
  u_row&lt;-which(U_lookup == user)[1]
  m_row&lt;-which(M_lookup==movie)

  knn_match&lt;- knn_txt[u_row,]

  knn_match&lt;-as.numeric(unlist(knn_match))

  dfm_mov&lt;- dfm[,m_row]

  dfm_test&lt;- dfm_mov[knn_match]

  c&lt;-mean(dfm_test[dfm_test!=0])
  c1&lt;-mean(dfm_mov[dfm_mov!=0])
  ifelse(c!="NaN",c,c1)
}
test&lt;-sapply(1:nrow(probe),
             function(x) mean_rating(probe[x,]))


The inputs to this function include the following.
Sparse matrix dfm

library(Matrix)
    dput(dfm)
new("dgTMatrix"
    , i = c(0L, 1L, 2L, 4L, 5L, 6L, 8L, 0L, 1L, 2L, 3L, 4L, 6L, 7L, 8L, 
0L, 2L, 3L, 6L, 7L, 8L, 1L, 2L, 4L, 5L, 6L, 7L, 8L, 9L, 0L, 1L, 
3L, 4L, 5L, 6L, 7L, 8L, 9L, 0L, 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 
9L, 0L, 1L, 3L, 4L, 6L, 7L, 8L, 9L, 0L, 2L, 3L, 5L, 6L, 7L, 9L, 
0L, 1L, 2L, 3L, 4L, 5L, 6L, 8L, 9L, 0L, 1L, 2L, 3L, 4L, 5L, 6L, 
7L, 9L)
    , j = c(0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
2L, 2L, 2L, 2L, 2L, 2L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 
8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 
9L, 9L)
    , Dim = c(10L, 10L)
    , Dimnames = list(NULL, NULL)
    , x = c(4, 3, 1, 2, 3, 1, 2, 1, 3, 3, 2, 3, 3, 3, 4, 2, 1, 2, 3, 2, 
1, 4, 1, 2, 2, 3, 2, 3, 4, 1, 4, 1, 3, 4, 3, 2, 2, 2, 4, 1, 2, 
2, 1, 2, 3, 1, 1, 1, 4, 1, 1, 2, 1, 1, 1, 4, 3, 3, 2, 1, 2, 2, 
1, 1, 3, 3, 4, 1, 2, 4, 2, 4, 1, 2, 2, 3, 4, 2, 1, 2, 4)
    , factors = list()
)


My probe dataset, these are the user movie combinations I am trying to predict for.

dput(probe)
structure(list(X = c(1145185L, 951920L, 1137277L, 180365L, 353195L
), movie = c(1L, 100L, 10000L, 10002L, 10004L), user = c(10L, 
1000004L, 1000033L, 1000035L, 1000053L), Rating = c(4L, 4L, 3L, 
5L, 4L)), .Names = c("X", "movie", "user", "Rating"), row.names = c("1145185", 
"951920", "1137277", "180365", "353195"), class = "data.frame")


My U_lookup table, this is where I convert from the actual id of a user to the row number of the matrix they are in.

dput(U_lookup[1:10,])
c(10L, 100000L, 1000004L, 1000027L, 1000033L, 1000035L, 1000038L, 
1000051L, 1000053L, 1000057L)


My M_lookup table, this is where I convert from the actual id of a movie to the row number of the matrix they are in.

dput(M_lookup[1:10,])
c(1L, 10L, 100L, 1000L, 10000L, 10001L, 10002L, 10003L, 10004L, 
10005L)


knn_txt is where I'm storing all the users nearest neighbors. Dfm and knn_text would have the same user in every row. I.e., row 1 in dfm would equate to the same user that is in row 1 of knn_text.

dput(knn_txt)
structure(list(V1 = c(1, 2, 1, 7, 5, 3, 2, 9, 1, 3), V2 = c(7, 
9, 5, 5, 3, 4, 2, 7, 4, 9), V3 = c(2, 9, 6, 3, 8, 9, 4, 7, 1, 
6)), .Names = c("V1", "V2", "V3"), row.names = c(NA, -10L), class = "data.frame")


Edit: As I have done some more testing myself it does seem like a lot of the time savings could come from more efficiently subsetting my large sparse matrix, dfm. I am planning on trying some different formats of the matrix, in addition to maybe looking at looping rather than using sapply (just because then I could say for user i, if user i=i-1 then I don't need to re-subset the matrix).
So after a lot of testing I found something that significantly reduces calculation time. Converting the format of my dfm to a different sparse format saves significant time when subsetting my main matrix (dfm).

dfm&lt;-as(dfm,"dgCMatrix")


This took my calculation time from approximately 11-12 seconds per line, to .2 seconds per line. As I understand it, this is because dgCMatrix format is optimized for subsetting by columns/column operations which is the first subsetting operation in my function. If you were wanting to subset by rows the optimal format would be RsparseMatrix, optimized for rows.

dfm&lt;-as(dfm,"RsparseMatrix")