Filter proteins by domain with InterProScan
1. Limit the InterPro scan to a specific database:
As you said you know the domain you are interested in so you know which database this is in then you can limit you InterPro scan to that database, which whill save a lot of time:

Then only select the database in which the domain is stored (e.g. ProSite)






2. If your domain is in the ProSite database (which it probably is) you can use 
ScanProsite and limit you search to a specific domain. 
You do this by going to the above link and select the last option:

Then, if you scroll a little bit down you can enter a ProSite accesion code of the domain/profile you are interested in:




3. Use PFAM scan

I have never used this before, so I don't know whether this would be faster than limiting the InterPro scan to the PFAMa database. You can of course check this out.



4. Clustering your sequences first

This will only work if your sequences look like each other. Depending on the size of the domain and the location in the sequences you can cluster all your sequences first (using for example CD-HIT). And then choose a representative sequences for each cluster and use these to subject to your analysis. For example when you know your domain is always at the end of the sequence then you trim the first part of each sequence and cluster the proteins over the whole length at 100% identity. If you then determine whether each cluster contains the domain or not (based on the representative sequence, which CD-HIT will provide you with) then you can assign these results to the sequences making up each cluster. But of course this will only reduce the search space whenever your sequences look like each other (or if parts do)



5. MOTIF search using downloaded profile or HMM

MOTIF search allows you to query against serveral database, such as PFAM, CDD, PROSITE etc. More interesting, it also allows you to specifiy your own profile or HMM. 

So you can download the HMM of your desired domain from e.g. PFAM  or the domain profile from Prosite.



6. Downloading HMM and use HMMscan (or HMMER)
This was suggested by @seven11, you can search your HMM domain in the PFAM database and then use HMMscan. 

Conclusion



If you are really interested in one domain. I think downloading the ProSite scan tool would be your best option. Notably, all webtools I mentioned above are available as command line tool, and that is what you need when having that amount of sequences. Back to ProSite scan, you can download this here and don't forget the README. As mentioned in the readme:

Input/Output:
  -e : specify the ID or AC of an entry in sequence-file
  -o : specify an output format : $formats_string
  -d : specify a prosite.dat file
  -p : specify a pattern or the AC of a prosite motif
  -f : specify a motif AC to scan against together with all its 
       releated post-processing motifs (but show only specified 
       motif hits)


You can use the -p to specify the acc of the pattern you want.

I'm quite busy so I typed this one really fast, please let me know when something is not clear.