Using text in one file to search for match in second file
Question: I'm using python 2.6 on linux.

I have two text files first.txt has a single string of text on each line. So
it looks like

lorem  
ipus  
asfd

The second file doesn't quite have the same format. it would look more like
this

1231 lorem  
1311 assss 31 1  
etc

I want to take each line of text from first.txt and determine if there's a
match in the second text. If there isn't a match then I would like to save the
missing text to a third file. I would like to ignore case but not completely
necessary. This is why I was looking at regex but didn't have much luck.

So I'm opening the files, using readlines() to create a list.  
Iterating through the lists and printing out the matches.

Here's my code

    
    
    first_file=open('first.txt', "r")
    first=first_file.readlines()
    first_file.close()
    
    second_file=open('second.txt',"r")
    second=second_file.readlines()
    second_file.close()
    
    while i < len(first):
      j=search[i]
      while k < len(second):
       m=compare[k]
       if not j.find(m):
        print m
       i=i+1
       k=k+1
    exit() 
    

It's definitely not elegant. Anyone have suggestions how to fix this or a
better solution?


Answer: My approach is this: Read the second file, convert it into lowercase and then
create a list of the words it contains. Then convert this list into a
[set](http://docs.python.org/library/stdtypes.html#set), for better
performance with large files.

Then go through each line in the first file, and if it (also converted to
lowercase, and with extra whitespace removed) is not in the set we created,
write it to the third file.

    
    
    with open("second.txt") as second_file:
        second_values = set(second_file.read().lower().split())
    
    with open("first.txt") as first_file:
        with open("third.txt", "wt") as third_file:
            for line in first_file:
                if line.lower().strip() not in second_values:
                    third_file.write(line + "\n")
    

set objects are a simple container type that is unordered and cannot contain
duplicate value. It is designed to allow you to quickly add or remove items,
or tell if an item is already in the set.

`with` statements are a convenient way to ensure that a file is closed, even
if an exception occurs. They are enabled by default from Python 2.6 onwards,
in Python 2.5 they require that you put the line `from __future__ import
with_statements` at the top of your file.

The `in` operator does what it sounds like: tell you if a value can be found
in a collection. When used with a list it just iterates through, like your
code does, but when used with a set object it uses hashes to perform much
faster. `not in` does the opposite. (Possible point of confusion: `in` is also
used when defining a `for` loop (`for x in [1, 2, 3]`), but this is
unrelated.)

