title: How to remove duplicate lines from a large text file efficiently?
body: <p>I want to edit a text as each line exists once in it. Each lines contains constantly 10 characters. I am generally  working on 5-6 million of lines. So the code i am using currently is consuming too much RAM. </p>
  
  <p>My code:</p>
  
  <pre><code>File.WriteAllLines(targetpath, File.ReadAllLines(sourcepath).Distinct())
  </code></pre>
  
  <p>So how can I make it less RAM consumer and less time-consumer at the same time?</p>