How do I find identical sequences in a FASTA file? I want to create a database for a proteomics study. Therefore, the mapping from a given sequence to a protein ID has to be unique. I am wondering whether there is already a built-in function in Biopython for that, but I could not find any. The duplicated sequences should be merged into a single entry and there respective IDs should be just concatenated (see expected outcome below).

Using the following example input

&gt;prot_id_1
MAWIGLDISKLFVENSRDAAA
&gt;prot_id_2
MENVSRIWRYQQRRVLSRAFTHWYLMGLTKHNHPS
&gt;prot_id_3
MRTRPSRDFHYIVGRITRESPEEKGIFYVFH
&gt;prot_id_4
MEMVLSSANPSTTFADSYVV
&gt;prot_id_5
MAWIGLDISKLFVENSRDAAA
&gt;prot_id_6
MENVSRIWRYQQRRVLSRAFTHWYLMGLTKHNHPS
&gt;prot_id_7
MAWIGLDISKLFVENSRDAAA


I currently do it as follows:

from Bio import SeqIO
import numpy as np
from Bio.Seq import Seq
from Bio.SeqRecord import SeqRecord
from Bio.Alphabet import IUPAC

# Read entries into dictionary, gi: sequence
record_dict = SeqIO.to_dict(SeqIO.parse("test.fasta", "fasta"))

# Extract sequence as string
record_dict = {gi: str(record_dict[gi].seq) for gi in record_dict}

# Check for duplicated sequences
seq_id_dict = {}
for k, v in record_dict.items():
    seq_id_dict.setdefault(v, []).append(k)

# Filter sequences that appear more than once
filter_dict = {seqi: li for seqi, li in seq_id_dict.items() if len(li) &gt; 1}

# Merge identifiers to a single one
merged_ids = {"_".join(sorted(li)): seqi for seqi, li in filter_dict.items()}

# Remove all entries that are associated with duplicate IDs
for gi in set(np.concatenate(filter_dict.values())):
    record_dict.pop(gi, None)

# Add the merged IDs again
record_dict.update(merged_ids)

# Create list of Seq objects
final_seq = [SeqRecord(Seq(seqi, IUPAC.protein), id=gi, name='', description='') for gi, seqi in record_dict.items()]

# Write file
SeqIO.write(final_seq, 'test_output.fasta', 'fasta')


which gives me the desired output:

&gt;prot_id_4
MEMVLSSANPSTTFADSYVV
&gt;prot_id_1_prot_id_5_prot_id_7
MAWIGLDISKLFVENSRDAAA
&gt;prot_id_3
MRTRPSRDFHYIVGRITRESPEEKGIFYVFH
&gt;prot_id_2_prot_id_6
MENVSRIWRYQQRRVLSRAFTHWYLMGLTKHNHPS


Is there a smarter way of accomplishing this?
The trick would be to swap the key in the dictionary to be the sequence itself.

Also I would recommend using a different separator that "_" since that is what the current ids have so that you can easily separate the individual ids from the concatenated id. I used a pipe "|" in this example. Also I just manually wrote the FASTA output instead of using Biopython to do it.

from Bio import SeqIO
from collections import defaultdict

dedup_records = defaultdict(list)
for record in SeqIO.parse("test.fasta", "fasta"):
    # Use the sequence as the key and then have a list of id's as the value
    dedup_records[str(record.seq)].append(record.id)
with open("Output.fasta", 'w') as output:
    for seq, ids in dedup_records.items():
        # Join the ids and write them out as the fasta
        output.write("&gt;{}\n".format('|'.join(ids)))
        output.write(seq + "\n")