title: sed optimization (large file modification based on smaller dataset)
body: <p>I do have to deal with very large plain text files (over 10 gigabytes, yeah I know it depends  what we should call large), with very long lines.</p>
  
  <p>My most recent task involves some line editing based on data from another file.</p>
  
  <p>The data file (which should be modified) contains 1500000 lines, each of them are e.g. 800 chars long. Each line is unique, and contains only one identity number, each identity number is unique)</p>
  
  <p>The modifier file is e.g. 1800 lines long, contains an identity number, and an amount and a date which should be modified in the data file.</p>
  
  <p>I just transformed (with Vim regex) the modifier file to sed, but it's very inefficient.</p>
  
  <p>Let's say I have a line like this in the data file:</p>
  
  <pre><code>(some 500 character)id_number(some 300 character)
  </code></pre>
  
  <p>And I need to modify data in the 300 char part.</p>
  
  <p>Based on the modifier file, I come up with sed lines like this:</p>
  
  <pre><code>/id_number/ s/^\(.\{650\}\).\{20\}/\1CHANGED_AMOUNT_AND_DATA/
  </code></pre>
  
  <p>So I have 1800 lines like this.</p>
  
  <p>But I know, that even on a very fast server, if I do a </p>
  
  <pre><code>sed -i.bak -f modifier.sed data.file
  </code></pre>
  
  <p>It's very slow, because it has to read every pattern x every line.</p>
  
  <p>Isn't there a better way?</p>
  
  <p><strong>Note:</strong> I'm not a programmer, had never learnt (in school) about algorithms.
  I can use awk, sed, an outdated version of perl on the server.</p>