E value cutoff for RNASeq search
Going by an e-value or bitscore cutoff will give you possible homologs, but it sound like you want to remove redundant sequences. If you want to cluster and combine similar sequences to make a smaller database, you could just go via sequence identity using something like CD-HIT. This, for example, is what's done to produce the UniRef set from the UniProt database.