title: remove duplicate lines from a file
body: <p>I have the following data:</p>
  
  <pre><code> number1
   I am writing line1 .
   number2
   First line .
   number3
   I am writing line2.
   number4
   Second line .
   number5
   I am writing line3 .
   number6
   Third line.
   number7
   I am writing line2 .
   number8
   Fourth line .
   number9
   I am writing line5 .
   number10
   Fifth line .
  </code></pre>
  
  <p>Now I want to remove the duplicate lines from this text file -- along with this I want to remove 1 preceding and 2 succeeding lines of the duplicate line. Such that after removal my data looks like:</p>
  
  <pre><code> number1
   I am writing line1 .
   number2
   First line .
   number3
   I am writing line2.
   number4
   Second line .
   number5
   I am writing line3 .
   number6
   Third line.
   number9
   I am writing line5 .
   number10
   Fifth line .
  </code></pre>
  
  <p>The size of my file is 60 GB and I am using a server with 64 GB RAM. I am using the following code for removing the duplicates:</p>
  
  <pre><code>fOutput = open('myfile','w')
  
  table_size = 2**16
  seen = [False]*table_size
  infile = open('test.ttl', 'r')
  while True:
      inFileLine1=infile.readline()
      if not inFileLine1:
          break #EOF
      inFileLine2=infile.readline()
      inFileLine3=infile.readline()
      inFileLine4=infile.readline()
      h = hash(inFileLine2) % table_size
      if seen[h]:
          dup = False
          with open('test.ttl','r') as f:
              for line1 in f:
                  if inFileLine2 == line1:
                      dup = True
                      break
              if not dup:
                  fOutput.write(inFileLine1)
                  fOutput.write(inFileLine2)
                  fOutput.write(inFileLine3)
                  fOutput.write(inFileLine4)
      else:
          seen[h] = True
          fOutput.write(inFileLine1)
          fOutput.write(inFileLine2)
          fOutput.write(inFileLine3)
          fOutput.write(inFileLine4)
  
  fOutput.close()
  </code></pre>
  
  <p>However, it turns out this code is very slow. Is there some way by which I may improve the efficiency of the code using parallelization  i.e. using all 24 cores available to me on my system or using any other technique.</p>
  
  <p>Although the above code is written in python -- but I am fine with efficient solutions in c++ or python or Java or using linux commands</p>
  
  <p>Here test.ttl is my input file with size 60GB</p>