title: Optimise the removal of duplicate lines from a large file
body: <p>How can I optimise this code. I want to remove a duplicate lines from a file but think using the set if inefficient and will limit the size of file I can parse.</p>
  
  <pre><code>file = open("sample.txt")
  
  output = []
  alreadyseen = set()    
  
  while True:
      lines = file.readlines(100000)
      if not lines:
          break
      for line in lines:
          pass # do something
          if line not in alreadyseen:
              output.append(line)
              alreadyseen.add(line)
      print(output)   
  </code></pre>