Finding common phrases between files with millions of lines
<p>I have two files  with the following number of lines : </p>

<pre><code>file1 - 110433003
file2 - 4838810
</code></pre>

<p>I need to find common phrases between these . Each line is of the form : </p>

<p>p1 ||| p2 ||| .......</p>

<p>The p1 of file1 can be the p2 in file2. Unfortunately, the code I have written is taking way too long to do this.  </p>

<pre><code>import sys
import os

if(len(sys.argv)&lt;2):
        print 'python CommonPhrases.py enFr hrEn commonFile'
        sys.exit()
enFr = open(sys.argv[1],'r')
hrEn = open(sys.argv[2],'r')
common = open(sys.argv[3],'w')
sethrEn = set([])
setenFr= set([])
for line in hrEn:
        englishPhrase = line.split(' ||| ')[1]
        sethrEn.add(englishPhrase)

for line in enFr:
        englishPhrase = line.split(' ||| ')[0]
        if(englishPhrase in sethrEn):
                common.write(englishPhrase+'\n')
</code></pre>

<p>Is there a faster way to do this ? </p>

<p>Thanks </p>
