How to refine this distribution comparison script My data look like this

They are compound similarity estimations(the whole file is around 10GB). What I am trying to achieve is to compare the similarity distributions of each compound using the Kolmogorov-Smirnoff test so as to use the data for AIC calculation.
This is my code:
import pandas as pd
from scipy.stats import ks_2samp
import re

with open('ecfp4_file.csv', 'r') as f, open('Metrics.tsv', 'a') as f_out:
    f_out.write('compound_1' + '\t' + 'compound_2' + '\t' + 'Similarity' + '\t' + 'KS Distance' + '\n')
    df = pd.read_csv(f, delimiter = ',', lineterminator = '\n', header = 0)

    for m in range(10, 99):
        dc1 = []
        dc2 = []
        m1 = m/100
        m2 = m/100+0.009
        df2 = df.loc[(df['ECFP4'] &gt;= m1) &amp; (df['ECFP4'] &lt; m2)]          #Work on different thresholds each time
    
        for i in range(0, df2.shape[0]):
            if len(df2.index[df2['id1'] == df2.iloc[i, 0]].tolist()) &gt;= 5:
                dc1.append(df2.iloc[i, 0])
                dc2.append(df2.iloc[i, 1])
        for i in range(0, len(dc1)):
            dc1_l = []
            dc2_l = []
            df3 = df.loc[df['id1'] == dc1[i]]
            dc1_l = df3.iloc[:, 2].tolist()
            df4 = df.loc[df['id1'] == dc2[i]]
            dc2_l = df4.iloc[:, 2].tolist()
            x1 = re.findall(r&quot;statistic=(.*)\,.*$&quot;, str(ks_2samp(dc1_l, dc2_l)))
            f_out.write(str(dc1[i]) + '\t' + str(dc2[i]) + '\t' + str(m1) + '\t' + str(x1[0]) + '\n')

which so far has taken 10 hours to run and it has not yet passed the 0.1 threshold, so it is highly unpractical. Does anyone know of a better way to do this? Or can help me refine my code for faster and better results?
I also tried to refine it and run it in R, but I get a vector memory exhausted (limit reached) error when I run it
install.packages(&quot;ggplot2&quot;)
library(ggplot2)
PS22_ECFP4 &lt;- read.csv(&quot;./PS22_ECFP4.csv&quot;)

df &lt;- data.frame(matrix(ncol = 4, nrow = 0))
cols &lt;- c(&quot;id1&quot;, &quot;id2&quot;, &quot;Similarity&quot;, &quot;KS Distance&quot;)
colnames(df) &lt;- cols

for (i in PS22_ECFP4) {
  sim_df &lt;- PS22_ECFP4[PS22_ECFP4$id1 == i,]
  dist1 &lt;- sim_df[ , 3]
  for (k in 1:nrow(sim_df)) {
    sim &lt;- sim_df[k , 3]
    sim_r &lt;- floor(sim * 100) / 100
    comp2 &lt;- sim_df[k, 2]
    comp2_df &lt;- PS22_ECFP4[PS22_ECFP4$id1==comp2,]
    dist2 &lt;- comp2_df[ , 3]
    if (length(dist1) &gt; 1 &amp; length(dist2) &gt; 1) {
      ks &lt;- ks.test(dist1, dist2)
      df[nrow(df) + 1,] = c(i, comp2, sim_r, ks$statistic)
    }
  }
}

write.csv(df, &quot;./Metrics.csv&quot;, row.names = FALSE)
I am not sure how useful this answer is. But may be you could try this.
For your python script:
You use f_out.write within a loop. I think usually this I/O operation is resource intensive. I guess the speed would improve significantly if you store the string in a variable and write the output only once outside the for loop like this:
out_str = ''
for i in range(0, len(dc1)):
            dc1_l = []
            dc2_l = []
            df3 = df.loc[df['id1'] == dc1[i]]
            dc1_l = df3.iloc[:, 2].tolist()
            df4 = df.loc[df['id1'] == dc2[i]]
            dc2_l = df4.iloc[:, 2].tolist()
            x1 = re.findall(r&quot;statistic=(.*)\,.*$&quot;, str(ks_2samp(dc1_l, dc2_l)))
            out_str +=  (str(dc1[i]) + '\t' + str(dc2[i]) + '\t' + str(m1) + '\t' + str(x1[0]) + '\n')

f_out.write(out_str)

Regarding the memory error in your R script, are you using 32bit build of R?
Please see this:
http://web.mit.edu/~r/current/arch/i386_linux26/lib/R/library/base/html/Memory-limits.html#
They say &quot;Under most 64-bit versions of Windows the limit for a 32-bit build of R is 4Gb: for the oldest ones it is 2Gb. The limit for a 64-bit build of R (imposed by the OS) is 8Tb.&quot;
Conclusion: Please try using your python script, but use the write operation outside the loop.