Convert population allele count into population allele frequency by using Perl or Python (translate available Bash into Perl or Python) I already know the Bash AWK solution and also R code for the question that I am asking. But, my file is so big and with R it takes very long time and I am afraid of AWK's mistake for this computation, so I wonder if there is a Perl or Python solution for my problem.

I have a population allele count data, and it looks like this:

1   0   0   0   0   0   0   0   0   0   1   2   1   0   0   0   0
0   2   0   0   0   0   0   0   0   0   0   4   0   2   0   0   0
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
2   2   0   0   2   1   0   0   0   0   2   4   0   0   0   2   0


Columns are populations and rows are SNPs. I have two rows per SNP (one row for the number of copies of allele "A" in each population and one row for the number of copies of allele "a").

In the example above, the first and second rows are number of alleles for the SNP1, the third and fourth rows are alleles for SNP 2, and like this until SNP20000000.

I want to calculate population allele frequency per SNP for all populations: frequency of allele A at SNP1 in population 1 = number of copies of allele A in population/total number of A/a gene copies in population and frequency of allele a at SNP1 in population 1 = number of copies of allele a in population/total number of A/a gene copies in the population. This means for each SNP number of copies of each allele should be divided to sum of the number of copies of allele "A" and "a" for each population. This is my desired output:

1   0   0   0   0   0   0   0   0   0   1   0.333333    1   0   0   0   0
0   1   0   0   0   0   0   0   0   0   0   0.666667    0   1   0   0   0
0   0   0   0   0   0   0   0   0   0   0   0           0   0   0   0   0
1   1   0   0   1   1   0   0   0   0   1   1           0   0   0   1   0


As I said, I have an R and Bash solution, but is there a way to do this estimation in Perl or Python?

Here is the Bash solution, but I do not know how to translate it into Perl or Python? I am learning Perl...

awk '{for(i = 1; i &lt;= NF; i++)
    tally[i] += $i};
    (NR%2) == 1
    {for(i=1; i&lt;=NF; i++)
        allele1[i] = $i};
        (NR%2) == 0
        {for(i = 1; i &lt;= NF; i++)
        allele2[i] = $i;
            for(i = 1; i &lt;= NF; i++)
                if(tally[i] == 0)
                    tally[i] = 1;
                for(i = 1; i &lt;= NF; i++)
                    printf allele1[i]/tally[i]"\t";
                    printf "\n";
                for(i = 1; i &lt;= NF; i++)
                    printf allele2[i]/tally[i]"\t";
                    printf "\n";
                for(i = 1; i &lt;= NF; i++)
                    tally[i]=0}' MyData |
sed 's/\t$//g'
I don't know how fast you need it to be, but this R solution runs on 20 million SNPs in under 2 min on my laptop:

allele_counts &lt;-matrix(rep(c(1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 1, 0, 0, 0, 0,
                     0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 2, 0, 0, 0,
                     0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
                     2, 2, 0, 0, 2, 1, 0, 0, 0, 0, 2, 4, 0, 0, 0, 2, 0), 10000000), 
                   ncol=17, byrow=TRUE)
allele1&lt;-allele_counts[ c(TRUE,FALSE), ]
allele2&lt;-allele_counts[ c(FALSE,TRUE), ]
total_counts &lt;- allele1+allele2
allele_counts[c(TRUE,FALSE), ] &lt;-allele1/total_counts
allele_counts[c(FALSE,TRUE), ] &lt;-allele2/total_counts


Edit

I decided to a head-to-head test of this solution vs. bli's pandas solution and your awk solution

I used readr for file IO:

library(readr)
allele_counts&lt;-as.matrix(read_tsv("allele_counts.txt", col_names=F))
...
write_tsv(as.data.frame(allele_counts), "allele_freq_R.txt", col_names=F)


Here are the timings:

cat Benchmarks/R.txt 
s       h:m:s   max_rss max_vms max_uss max_pss io_in   io_out  mean_load
103.7556        0:01:43 2211.16 2518.59 2209.11 2209.15 0.00    27.58   73.62

cat Benchmarks/Python.txt 
s       h:m:s   max_rss max_vms max_uss max_pss io_in   io_out  mean_load
132.6425        0:02:12 4872.45 6042.28 4870.89 4870.92 2.67    46.59   99.92
cat Benchmarks/awk.txt 
s       h:m:s   max_rss max_vms max_uss max_pss io_in   io_out  mean_load
126.4455        0:02:06 4.22    413.55  0.93    0.97    0.00    130.68  91.28


The R solution is slightly faster than the pandas solution and uses half as much memory. Your awk one-liner is about the same speed as the pandas script, but requires a tiny fraction of the memory footprint.