delete duplicate word combinations in textfile with python
<p>with the help of eumiro <a href="http://stackoverflow.com/questions/12815591/delete-duplicate-rows-in-textfile-except-it-contains-a-or">Delete duplicate rows in textfile - except it contains a &quot;{&quot; or &quot;}&quot;</a>
i could successfully remove duplicate line in a large textfile. That's a huge step from 60MB to 3MB textfile.</p>

<p>But now i want delete duplicate words like this:</p>

<pre class="lang-none prettyprint-override"><code>  @INBOOK{Miller1992,
  author = {Miller, Rowland S. und Mark R. Leary and Miller, Rowland S. und Mark
    R. Leary and Miller, Rowland S. und Mark R. Leary and Miller, Rowland
    S. und Mark R. Leary and Miller, Rowland S. und Mark R. Leary and
    Miller, Rowland S. und Mark R. Leary and Miller, Rowland S. und Mark
    Miller, Rowland S. und Mark R. Leary},
  year = {1992},
  editor = {Teun A. van Dijk and Teun A. van Dijk and Teun A. van Dijk and Teun
    A. van Dijk and Teun A. van Dijk and Teun A. van Dijk and Teun A.
    van Dijk and Teun A. van Dijk and Teun A. van Dijk and Teun A. van
    Dijk and Teun A. van Dijk and Teun A. van Dijk and Teun A. van Dijk
    and Teun A. van Dijk and Teun A. van Dijk and Teun A. van Dijk and
    Teun A. van Dijk and Teun A. van Dijk and Teun A. van Dijk and Teun
    and Teun A. van Dijk and Teun A. van Dijk and Teun A. van Dijk},
  title = {Handbook of discourse analysis (Bd. 3/4)},
</code></pre>

<p>the result should look like this:</p>

<pre class="lang-none prettyprint-override"><code>  @INBOOK{Miller1992,
  author = {Miller,  Rowland S. und Mark R. Leary},
  year = {1992},
  editor = {Teun A. van Dijk},
  title = {Handbook of discourse analysis (Bd. 3/4)},
</code></pre>

<p>The textfile has 70000 Lines and the authornames could be used in multiple entries. Therefor only the duplicates between the curly brackets (over multiple lines) should removed:</p>

<pre class="lang-none prettyprint-override"><code>  author = {Miller, Rowland S. und Mark R. Leary and Miller, Rowland S. und Mark
  R. Leary and Miller, Rowland S. und Mark R. Leary and Miller, Rowland
  S. und Mark R. Leary and Miller, Rowland S. und Mark R. Leary and
  Miller, Rowland S. und Mark R. Leary and Miller, Rowland S. und Mark
  Miller, Rowland S. und Mark R. Leary},
</code></pre>

<p>I tried to modify my Python-Skript which delete duplicate lines to delete duplicate words between the curly brackets but i'm stucked:</p>

<pre class="lang-py prettyprint-override"><code>words_seen = set() # holds words already seen 
outfile = open("literatur_clean.txt", "w") 
for line in open("literatur_dupl.txt", "r"): 
    if ('{' in line or '}' in line
        # some code to check whether the words are duplicate
outfile.close() 
</code></pre>
