Python compare every line in file with itself
On reading the file twice

Constructing data structures is costly. You do it once for each line in the file (\$n\$ times) and then do it again \$n^2\$ times. There is absolutely no need as you could only use one list to do so:

&gt;&gt;&gt; a = [[1], [2], [3]]
&gt;&gt;&gt; for i in a:
...     for j in a:
...         print(i, j)
...
[1] [1]
[1] [2]
[1] [3]
[2] [1]
[2] [2]
[2] [3]
[3] [1]
[3] [2]
[3] [3]


That way, with only one list, you avoid spending time converting strings to datastructure and can focus on your algorithm.

So just build the list once and use it any time you want.

Use functions

It lets you reuse your work more easily, by changing parameters. You can also more easily add code to handle command-line arguments and not need to change your code each time you change your input/output file.

A simple layout could look like:

def parse_input(filename):
    ...

def compute_output(output_file, data, filter_value):
    ...

def filter_file(path, filter_value=3, in_name='file', out_name='filteredSets'):
    data = parse_input('{}/{}'.format(path, in_name))
    with open('{}/{}{}'.format(path, out_name, filter_value), 'w') as out_file:
        if len(data) &gt; 1:
            compute_output(out_file, data, filter_value)
        else:
            for datum in data:
                print(datum, file=out_file)


if __name__ == '__main__':
    filter_file(arg2 + arg1)


I’m assuming Python 3 here for the print(datum, file=out_file) line. Change it accordingly if you are using Python 2. It, however, will change your output a bit, but more on that later. I'm also using a for loop in the else just in case there is no rows in data.

Separating concerns with functions let you focus on being efficient for simple tasks. And, if later you want to improve your code thanks to multiprocessing, it will be easier to process it in parallel (using map_async for instance).

Iterate directly over items

Rather than iterating over indices and using these indices to retrieve items. Instead of:

for i in range(0, len(list1)):
    ...
    set2 = list1[i]


You should use:

for set2 in list1:


It's both easier to read and faster. If you truly need the index, you can still use enumerate:

for i, set2 in enumerate(list1):


But as far as eliminating tests against the same line in our product you can test if the object is the same:

for set1 in list1:
    for set2 in list1:
        if set1 is not set2:
            # Do something


Note that I used is instead of == to be sure that we are performing tests on the same line and not on lines with identical values:

&gt;&gt;&gt; a = [[1], [1], [1]]
&gt;&gt;&gt; for i in a:
...     for j in a:
...         print(i, j, i==j, i is j)
...
[1] [1] True True
[1] [1] True False
[1] [1] True False
[1] [1] True False
[1] [1] True True
[1] [1] True False
[1] [1] True False
[1] [1] True False
[1] [1] True True


Managing "stop-early" in loops

Python allows loops (for and while) an optional else clause. This clause executes only if the loop executed entirely without reaching a break (or a return, for what it's worth).

In your case, you could use that at your advantage in the inner for loop: instead of waiting to reach the end of the file for each line, you can break as soon as you find that two lines shares more than 3 items. If you handle writing lines in the else clause of the inner for loop, you can rely on that break to not print a line that has too many items in common with an other one.

Miscellaneous


You can use os.path.join to concatenate folders and filenames.
You can use set1 &amp; set2 as a shorthand to set1.intersection(set2).
You should write each import statement on its own line.


Proposed improvements

import os
from ast import literal_eval

def parse_input(filename):
    with open(filename) as f:
        data = [set(literal_eval(line)) for line in f]
    return data

def compute_output(output_file, data, filter_value):
    for set1 in data:
        for set2 in data:
            if set1 is set2:
                # Do not try to compare a row with itself
                continue
            if len(set1 &amp; set2) &gt;= filter_value:
                # Do not keep lines that have more than a few items
                # in common with any other line
                break
        else:
            # If the second for loop did not break, then keep the line
            print(set1, file=output_file)

def filter_file(path, filter_value=3, in_name='file', out_name='filteredSets'):
    data = parse_input(os.path.join(path, in_name))
    output_filename = os.path.join(path, '{}{}'.format(out_name, filter_value))
    with open(output_filename, 'w') as out_file:
        compute_output(out_file, data, filter_value)


if __name__ == '__main__':
    filter_file(arg2 + arg1)


You should have noticed that I changed the output format a bit. Now each line is a set instead of a list. It's easy to change that by calling print(list(set1), file=output_file) instead. It may change the order, but wont change the content. You could even use sorted(set1).

Beyond ast.literal_eval

I don't know if you can manage the input files of if you have to build around their limitations. But in case you can change them, I suggest you have a look at the pickle module.

Basicaly, when creating your input file, you can pickle.dump a list of all rows (possibly already a list of sets) and then, load them in this script using a simple data = pickle.load(os.path.join(arg2 + arg1, 'file')).