title: Searching for string in massive files efficiently
body: <p>I found variants of this idea, but none that could get me (very new to python) to where I need to be.</p>
  
  <p>Here is the scenario:</p>
  
  <ol>
  <li>I have one massive, 27 gig <code>hashfile.txt</code> consisting of unique strings all on separate lines.</li>
  <li>I need to parse this file line by line searching for a match in another, not-so-large (~800mb) <code>addresses.txt</code> file</li>
  <li>When a match is found, this needs to be written to <code>outfile.txt</code></li>
  </ol>
  
  <p>My current code has been optimized to the best of my ability, but only gets around 150 lines/second. Considering I have over 1.5 billion lines in my <code>hashfile.txt</code>, any optimization would help.</p>
  
  <pre><code>fin = 'hashed.txt'
  nonzeros = open('addrOnly.txt', 'r')
  fout = open('hits.txt', 'w')
  lines = nonzeros.read()
  i = 0
  count = 0
  
  with open(fin, 'r') as f:
      for privkey in f:
              address = privkey.split(", ")[0]
              if address in lines:
                      fout.write(privkey)
              i = i+1
              if i%100 == 0:
                      count = count + 100
                      print "Passed: " + str(count)
  </code></pre>