title: Removing Duplicate Words Across Multiple and Large Dictionary Files
body: <p>I have roughly ~600GB of dictionaries I've accumulated over the years, and I decided I want to clean them up and sort them</p>
  
  <p>First of all, each file on average is very large, anywhere from 500MB to 9GB in size. A prerequisite for what I want to do is that I sort each dictionary. My end goal is to entirely remove duplicate words <em>within</em> and <em>throughout</em> <strong>all</strong> dictionary files.</p>
  
  <p>The reason for this is that most of my dictionaries are sorted and organized by categories, but duplicates still often exist.</p>
  
  <pre><code>Load file
       Read each line and put into data structure
       Sort and remove any and all duplicate
  Load next file and repeat
  
  Once all files are individually unique, compare against eachother and remove duplicates
  </code></pre>
  
  <p><strong>For Dictionaries D{1} to D{N}:</strong></p>
  
  <p>1) Sort <em>D{1}</em> through <em>D{N}</em> individually. </p>
  
  <p>2) Check uniqueness of each word in <em>D{i}</em></p>
  
  <p>3) For each word in <em>D{i}</em>, check ALL words across <em>D{i+1}</em> to <em>D{N}</em>. Delete each word if unique in <em>D{i}</em> first.</p>
  
  <ul>
  <li>I am considering using a sort of "hash" to improve this algorithm. Possibly by only checking the first one or two characters, since the list will be sorted (e.g. hash beginning line location for words starting with a, b, etc.).</li>
  </ul>
  
  <p>4) Save and exit. </p>
  
  <p>Example before (but far smaller):</p>
  
  <pre><code>    Dictionary 1            Dictionary 2            Dictionary 3
  
      ]a                      0u3TGNdB                2 KLOCK
      all                     avisskriveri            4BZ32nKEMiqEaT7z
      ast                     chorion                 4BZ5
      astn                    chowders                bebotch
      apiala                  chroma                  bebotch
      apiales                 louts                   bebotch
      avisskriveri            lowlander               chorion
      avisskriverier          namely                  PC-Based
      avisskriverierne        silking                 PC-Based
      avisskriving            underwater              PC-Based
  </code></pre>
  
  <p>So it would see avisskriveri, chorion, bebotch and PC-Based are words that repeate both within and among each of the three dictionaries. So I see avisskriveri in <em>D{1}</em> first, so remove it in all other instances that I have seen it in. Then I see chorion in <em>D{2}</em> first, and remove that in all other instances first, and so forth. In <em>D{3}</em> bebotch and PC-Based are replicated, so I want to delete all but one entry of it (unless I've seen it before). Then save all files and close.</p>
  
  <p>Example after:</p>
  
  <pre><code>     Dictionary 1           Dictionary 2            Dictionary 3
  
       ]a                     0u3TGNdB                2 KLOCK
       all                    chorion                 4BZ32nKEMiqEaT7z
       ast                    chowders                4BZ5
       astn                   chroma                  bebotch
       apiala                 louts                   PC-Based
       apiales                lowlander                   
       avisskriveri           namely              
       avisskriverier         silking                 
       avisskriverierne       underwater                          
       avisskriving 
  </code></pre>
  
  <p><strong>Remember:</strong> I do NOT want to create any new dictionaries, only remove duplicates across all dictionaries.</p>
  
  <p><strong>Options:</strong></p>
  
  <ul>
  <li><p>"Hash" the amount of unique words for each file, allowing the program to estimate the computation time.</p></li>
  <li><p>Specify a way give the location of the first word beginning with the desired first letter. So that the search may "jump" to a line and skip unecessary computational time.</p></li>
  <li><p>Run on GPU for high performance parallel computing. (This is an issue because getting the data off of the GPU is tricky)</p></li>
  </ul>
  
  <p><strong>Goal:</strong> Reduce computational time and space consumption so that the method is affordable on a standard machine or server with limited abilities. Or device a method for running it remotely on a GPU cluster.</p>
  
  <p>tl;dr - Sorting unique words across hundreds of files, where each file is 1-9GB in size.</p>