title: How to compare large text files?
body: <p>I have a general question on your opinion about my "technique".</p>
  
  <p>There are 2 textfiles (<code>file_1</code> and <code>file_2</code>) that need to be compared to each other. Both are very huge (3-4 gigabytes, from 30,000,000 to 45,000,000 lines each). 
  My idea is to read several lines (as many as possible) of <code>file_1</code> to the memory, then compare those to <em>all</em> lines of <code>file_2</code>. If there's a match, the lines from both files that match shall be written to a new file. Then go on with the next 1000 lines of <code>file_1</code> and also compare those to <em>all</em> lines of <code>file_2</code> until I went through <code>file_1</code> completely.</p>
  
  <p>But this sounds actually really, really time consuming and complicated to me. 
  Can you think of any other method to compare those two files?</p>
  
  <p>How long do you think the comparison could take?
  For my program, time does not matter that much. I have no experience in working with such huge files, therefore I have no idea how long this might take. It shouldn't take more than a day though. ;-) But I am afraid my technique could take forever...</p>
  
  <p>Antoher question that just came to my mind: how many lines would you read into the memory? As many as possible? Is there a way to determine the number of possible lines before actually trying it?
  I want to read as many as possible (because I think that's faster) but I've ran out of memory quite often.</p>
  
  <p>Thanks in advance.</p>
  
  <p><strong>EDIT</strong>
  I think I have to explain my problem a bit more.</p>
  
  <p>The purpose is not to see if the two files in general are identical (they are not). 
  There are some lines in each file that share the same "characteristic". 
  Here's an example:
  <code>file_1</code> looks somewhat like this:</p>
  
  <pre><code>mat1 1000 2000 TEXT      //this means the range is from 1000 - 2000
  mat1 2040 2050 TEXT
  mat3 10000 10010 TEXT
  mat2 20 500 TEXT
  </code></pre>
  
  <p><code>file_2</code>looks like this:</p>
  
  <pre><code>mat3 10009 TEXT
  mat3 200 TEXT
  mat1 999 TEXT
  </code></pre>
  
  <p><code>TEXT</code> refers to characters and digits that are of no interest for me, <code>mat</code> can go from <code>mat1 - mat50</code> and are in no order; also there can be 1000x <code>mat2</code> (but the numbers in the next column are different). I need to find the fitting lines in a way that: matX is the same in both compared lines an the number mentioned in <code>file_2</code> fits into the range mentioned in <code>file_1</code>.
  So in my example I would find one match: line 3 of <code>file_1</code>and line 1 of <code>file_2</code> (because both are mat3 and 10009 is between 10000 and 10010).
  I hope this makes it clear to you!</p>
  
  <p>So my question is: how would you search for the matching lines?</p>
  
  <p>Yes, I use Java as my programming language.</p>
  
  <p><strong>EDIT</strong>
  I now divided the huge files first so that I have no problems with being out of memory. I also think it is faster to compare (many) smaller files to each other than those two huge files. After that I can compare them the way I mentioned above. It may not be the perfect way, but I am still learning ;-) 
  Nonentheless all your approaches were very helpful to me, thank you for your replies!</p>