Filtering FASTQ file based on read names from other file (how to increase performance) Python I have some code here that basically takes in a list of IDs (sequencing read names) using STDIN, and then filters fin (a gzipped FASTQ file) to just those reads with a read name which matches one of the IDs, and then writes the name and sequence to fout (a gzipped FASTA file). Fairly standard data processing problem in bioinformatics.
The code below can run through about 100k/sec lines of fin, however fin is about 500M lines, so it still takes over an hour.
Can this be sped up significantly? I'm hoping for a 10x or better improvement in performance. I may be taking a performance hit running line.decode('utf-8'), however it's necessary for matching since gzip files are read as bytestring. I'm willing to drop down and rewrite this in C if this will substantially increase speed.
import sys
import gzip
import re

id_array = []
for line in sys.stdin:
    id_array.append(line.rstrip())

ids = set(id_array)

with gzip.open(sys.argv[1], &quot;r&quot;) as fin:
    with gzip.open(sys.argv[2], &quot;w&quot;) as fout:

        take_next_line = False
        for line in fin.readlines():
            seq_search = re.search('[@+]([^\s]+)', line.decode('utf-8'))

            if take_next_line:
                fout.write(line)
                take_next_line = False

            elif seq_search and seq_search.group(1) in ids:
                take_next_line = True
                fout.write('&gt;' + line[1:])
Do not reinvent the wheel. There are bioinformatics tools that accomplish this task.
To extract reads from fastq files by IDs, use seqtk subseq.

Extract sequences with names in file name.lst, one sequence name per line:
seqtk subseq in.fq name.lst &gt; out.fq

It works with fasta files as well.
Use conda install seqtk or conda create --name seqtk seqtk to install the seqtk package, which has other useful functionalities as well, and is very fast.
Alternatively, use BBMap filterbyname.sh, see docs:

By default, &quot;filterbyname&quot; discards reads with names in your name list, and keeps the rest. To include them and discard the others, do this:
filterbyname.sh in=003.fastq out=filter003.fq names=names003.txt include=t

SEE ALSO:
Extracting specific sequences from FASTQ using Seqtk
How To Extracting Fastq Sequence For Given Fastq Ids And Fastq File
How To Extract A Subset Of Reads In Fastq Using An Id List?