Remove Redundant Sequences from FASTA file in Python I'm attempting to remove redundant sequences from a fasta file (from NCBI). When I execute this code, it returns the number of spots, not the number of sequences. (Number of spots: 408,293, Number of sequences: 40,830)
Please let me know if you have run a code that can remove the duplicate ideas or have any ideas/suggestions.
Thank you SO much.
from Bio import SeqIO
import time

start = time.time() 

seen = []
records = []

for record in SeqIO.parse(&quot;b4r2.fasta&quot;, &quot;fasta&quot;):  
    if str(record.seq) not in seen:
        seen.append(str(record.seq))
        records.append(record)


#writing to a fasta file
SeqIO.write(records, &quot;no_dupes_b4r2.fasta&quot;, &quot;fasta&quot;)
end = time.time()

print(f&quot;Run time is {(end- start)/60}&quot;)
Your code (as currently formatted) should generate the correct output, but it is not efficient. Make seen a set to improve these O(n) in calls to O(1) time complexity- this will speed it up a lot
seen = set()
records = []

for record in SeqIO.parse(&quot;b4r2.fasta&quot;, &quot;fasta&quot;):  
    if record.seq not in seen:
        seen.add(record.seq)
        records.append(record)