A series of yes-no questions to predict yes-no outcomes I have been writing R scripts for about a week. I have been writing code for 20 years. I have a series of yes-no questions that I hope to predict yes-no outcomes so I know the R-squared will be terrible but as long as the variable is significant I am fine.

Since I am creating a number with my yes-no questions the order makes a difference and I have 14 questions which I plan to turn into 8-bit numbers (121,080,960 permutations). This is taking about 88 hours on my desktop. I am looking for anyplace to optimize my code. I have read about Multi core/thread SNOW but that is beyond me right now without some help. I also know I can compile functions but my code is short and only see one place to do so.

Please review my code, any help would be appreciated.

library(data.table)
ffName = "Perms8bit.csv"
# Read table of 1s and 0s
Bin_Matrix &lt;-read.table("LC_FR_F17BIN.csv",header=TRUE,sep=",")

Perm_Best &lt;- 0
holdR  &lt;- 0
s&lt;- Sys.time()
cnt &lt;-0

# Read the file 121,080,960 permutations to test
# Originally read the file in blocks but benchmark testing
rawData &lt;- fread(ffName, sep=",")

   for (cnt in 1:121080960){
        # create a table result x binary number
        Bin_num &lt;- data.table(result = c(Bin_Matrix[,15]),numb = c(2^0* (Bin_Matrix [,as.numeric(rawData[cnt,1])]) + 2^1*(Bin_Matrix [,as.numeric(rawData[cnt,2])]) +
2^2*(Bin_Matrix [,as.numeric(rawData[cnt,3])]) + 2^3*(Bin_Matrix [,as.numeric(rawData[cnt,4])]) +
2^4*(Bin_Matrix [,as.numeric(rawData[cnt,5])]) + 2^5*(Bin_Matrix [,as.numeric(rawData[cnt,6])]) +
2^6*(Bin_Matrix [,as.numeric(rawData[cnt,7])]) + 2^7*(Bin_Matrix [,as.numeric(rawData[cnt,8])])))

    #linear regession
    linearMod &lt;- lm(result ~ numb, data = Bin_num)
    ModSum &lt;- summary(linearMod)
    theR &lt;- ModSum$r.squared

    # Find the top R squared, result is 1s and 0s so R squared will not be high
    if (theR &gt; .0175) {
        holdR = theR
        Perm_Best &lt;- data.table(cnt = cnt,R = holdR,rawData[cnt,])

         #append a file with best fits
         write(toString(Perm_Best,nrows= 1),file = "RSQ.txt", append = TRUE)
      }


   }
print (Sys.time()-s) #benchmark
print(holdR)         # just to let me know its done
print (Perm_Best)


Bin_Matrix is a file with around 2,000 rows, header, and a bunch of 1s and 0s in csv form.  RawData is a  table with 121 million rows with the different permutations (examples:

1,2,3,4,5,6,7,8
1,2,3,4,5,6,7,9
...
14,1,2,3,4,5,6,7
14,1,2,3,4,5,6,8
Unfortunately, without the two csv files, we are not able to test your code... Maybe if you could show us the profiler output (see the example in ?summaryRprof) on a few iterations, it would help. What I suspect is that it will show most of the time is wasted either in the lm fit, or in the write call. Here are suggestions to improve on both as potential issues:

Alternative to lm

Since you are doing simple linear regression, i.e. regression with a single variable, there is a closed-form solution for the R-squared and it is fast to compute with R's built-ins:

RSQ1 &lt;- function(x, y) {
   mx  &lt;- mean(x)
   my  &lt;- mean(y)
   mxy &lt;- mean(x * y)
   mxx &lt;- mean(x * x)
   myy &lt;- mean(y * y)
  (mxy - mx * my)^2 / (mxx - mx^2) / (myy - my^2)
}


Compare this with your implementation

RSQ2 &lt;- function(z) {
  linearMod &lt;- lm(y ~ x, data = z)
  ModSum &lt;- summary(linearMod)
  ModSum$r.squared
}


Here I test both functions assuming your Bin_Matrix has 1000 rows. Your mileage may differ depending on how many observations you really have, something your question does not yet provide:

n &lt;- 1000
z &lt;- data.frame(x = runif(n), y = runif(n))

RSQ1(z$x, z$y)
# [1] 0.0007529693
RSQ2(z)
# [1] 0.0007529693

library(microbenchmark)
microbenchmark(RSQ1(z$x, z$y), RSQ2(z))
# Unit: microseconds
#            expr      min        lq      mean    median        uq      max neval
#  RSQ1(z$x, z$y)  111.707  120.8345  146.3035  126.5945  143.2855  689.371   100
#         RSQ2(z) 2531.381 2554.2105 2883.7659 2626.9020 2966.8550 5103.368   100


Inefficient write

The way you are using write is inefficient in that every time it is called, it must open the file, move the filehandle all the way to the end, write a line, then close the file. If your R squared is improved upon very often, this could start adding up. You would gain by doing the following.


Before your for loop, open a filehandle: fh &lt;- open("RSQ.txt")
Make write append to the filehandle: write(toString(Perm_Best, nrows= 1), file = fh)  (notice, there is no need for append = TRUE anymore)
After the for loop, close the file handle: close(fh)


Can you please give these suggestions a try and let me know if it helps? If it doesn't, maybe provide the output of the profiler? I hope it helps, good luck!



P.S.: Regarding coding style, you might find this approach for computation  Bin_num a little easier on the eye; it should not have a significant speed impact:

idx &lt;- as.numeric(rawData[cnt, 1:8])
bm  &lt;- Bin_Matrix[, idx]
Bin_num &lt;- data.table(result = c(Bin_Matrix[, 15]),
                      numb   = colSums(2^(0:7) * t(bm)))