python library for large tab/comma delimited text file
<p>I have some big genomic data files to analyze, which come in two forms, one individual dosage file like this:</p>

<pre><code>id                      snp1    snp2    snp3    snp4    snp5    snp6
RS1-&gt;1000001    DOSE    1.994   1.998   1.998   1.998   1.830   1.335
RS1-&gt;1000002    DOSE    1.291   1.998   1.998   1.998   1.830   1.335
RS1-&gt;100001     DOSE    1.992   1.998   1.998   1.998   1.830   1.335
RS1-&gt;100002     DOSE    1.394   1.998   1.998   1.998   1.830   1.335
RS1-&gt;10001      DOSE    1.994   1.998   1.998   1.998   1.830   1.335
RS1-&gt;1001001    DOSE    1.904   1.998   1.998   1.998   1.830   1.335
RS1-&gt;1002001    DOSE    1.094   1.998   1.998   1.998   1.830   1.335
RS1-&gt;1003001    DOSE    1.994   1.998   1.998   1.998   1.830   1.335
RS1-&gt;1004001    DOSE    1.994   1.998   1.998   1.998   1.830   1.335
RS1-&gt;1005002    DOSE    1.994   1.998   1.998   1.998   1.830   1.335
</code></pre>

<p>The other contains some summary info:</p>

<pre><code>SNP         Al1 Al2 Freq1   MAF     Quality Rsq 
22_16050607 G   A   0.99699 0.00301 0.99699 0.00000
22_16050650 C   T   0.99900 0.00100 0.99900 0.00000
22_16051065 G   A   0.99900 0.00100 0.99900 0.00000
22_16051134 A   G   0.99900 0.00100 0.99900 0.00000
rs62224609  T   C   0.91483 0.08517 0.91483 -0.00000
rs62224610  G   C   0.66733 0.33267 0.66733 0.00000
22_16051477 C   A   0.99399 0.00601 0.99399 -0.00000
22_16051493 G   A   0.99900 0.00100 0.99900 -0.00000
22_16051497 A   G   0.64529 0.35471 0.64529 0.00000
</code></pre>

<p>The SNP column in the second file corresponds the snp1, snp2... in the first file. I need to use the summary info in the second file to do some quality check and selection, then apply some statistical analysis on the data in the first file accordingly. </p>

<p>The question is, is there a python library suitable for this task? Performance is vital here, because these are really huge files. Thanks!</p>
