title: Performance of Writing to File C#
body: <p><strong>Overview of My Situation:</strong></p>
  
  <p>My task is to read strings from a file, and re-format them to a more useful format. After reformatting the input, I have to write it to a output file.</p>
  
  <p>Here is an Example of what has to be done.
  Example of File Line :</p>
  
  <pre><code>ANO=2010;CPF=17834368168;YEARS=2010;2009;2008;2007;2006 &lt;?xml version='1.0' encoding='ISO-8859-1'?&gt;&lt;QUERY&gt;&lt;RESTITUICAO&gt;&lt;CPF&gt;17834368168&lt;/CPF&gt;&lt;ANO&gt;2010&lt;/ANO&gt;&lt;SITUACAODECLARACAO&gt;Sua declaração não consta na base de dados da Receita Federal&lt;/SITUACAODECLARACAO&gt;&lt;DATACONSULTA&gt;05/01/2012&lt;/DATACONSULTA&gt;&lt;/RESTITUICAO&gt;&lt;RESTITUICAO&gt;&lt;CPF&gt;17834368168&lt;/CPF&gt;&lt;ANO&gt;2009&lt;/ANO&gt;&lt;SITUACAODECLARACAO&gt;Sua declaração não consta na base de dados da Receita Federal&lt;/SITUACAODECLARACAO&gt;&lt;DATACONSULTA&gt;05/01/2012&lt;/DATACONSULTA&gt;&lt;/RESTITUICAO&gt;&lt;RESTITUICAO&gt;&lt;CPF&gt;17834368168&lt;/CPF&gt;&lt;ANO&gt;2008&lt;/ANO&gt;&lt;SITUACAODECLARACAO&gt;Sua declaração não consta na base de dados da Receita Federal&lt;/SITUACAODECLARACAO&gt;&lt;DATACONSULTA&gt;05/01/2012&lt;/DATACONSULTA&gt;&lt;/RESTITUICAO&gt;&lt;RESTITUICAO&gt;&lt;CPF&gt;17834368168&lt;/CPF&gt;&lt;ANO&gt;2007&lt;/ANO&gt;&lt;SITUACAODECLARACAO&gt;Sua declaração consta como Pedido de Regularização(PR), na base de dados da Secretaria da Receita Federal do Brasil&lt;/SITUACAODECLARACAO&gt;&lt;DATACONSULTA&gt;05/01/2012&lt;/DATACONSULTA&gt;&lt;/RESTITUICAO&gt;&lt;RESTITUICAO&gt;&lt;CPF&gt;17834368168&lt;/CPF&gt;&lt;ANO&gt;2006&lt;/ANO&gt;&lt;SITUACAODECLARACAO&gt;Sua declaração não consta na base de dados da Receita Federal&lt;/SITUACAODECLARACAO&gt;&lt;DATACONSULTA&gt;05/01/2012&lt;/DATACONSULTA&gt;&lt;/RESTITUICAO&gt;&lt;STATUS&gt;&lt;RESULT&gt;TRUE&lt;/RESULT&gt;&lt;MESSAGE&gt;&lt;/MESSAGE&gt;&lt;/STATUS&gt;&lt;/QUERY&gt;
  </code></pre>
  
  <p>This input file has on each line two important informations: <code>CPF</code> which is the document number I will use, and the <code>XML</code> file (that represents the return of a query for the document on a database).</p>
  
  <p><strong>What I Must Achieve:</strong></p>
  
  <p>Each Document, in this <code>old format</code> has an <code>XML</code> containing the query returns for all the years (2006 to 2010). After reformatting it, each input line is converted to 5 output lines :</p>
  
  <pre><code>CPF=17834368168;YEARS=2010; &lt;?xml version='1.0' encoding='ISO-8859-1'?&gt;&lt;QUERY&gt;&lt;RESTITUICAO&gt;&lt;CPF&gt;17834368168&lt;/CPF&gt;&lt;ANO&gt;2010&lt;/ANO&gt;&lt;SITUACAODECLARACAO&gt;Sua declaração não consta na base de dados da Receita Federal&lt;/SITUACAODECLARACAO&gt;&lt;DATACONSULTA&gt;05/01/2012&lt;/DATACONSULTA&gt;&lt;/RESTITUICAO&gt;&lt;STATUS&gt;&lt;RESULT&gt;TRUE&lt;/RESULT&gt;&lt;MESSAGE&gt;&lt;/MESSAGE&gt;&lt;/STATUS&gt;&lt;/QUERY&gt;
  CPF=17834368168;YEARS=2009; &lt;?xml version='1.0' encoding='ISO-8859-1'?&gt;&lt;QUERY&gt;&lt;RESTITUICAO&gt;&lt;CPF&gt;17834368168&lt;/CPF&gt;&lt;ANO&gt;2009&lt;/ANO&gt;&lt;SITUACAODECLARACAO&gt;Sua declaração não consta na base de dados da Receita Federal&lt;/SITUACAODECLARACAO&gt;&lt;DATACONSULTA&gt;05/01/2012&lt;/DATACONSULTA&gt;&lt;/RESTITUICAO&gt;&lt;STATUS&gt;&lt;RESULT&gt;TRUE&lt;/RESULT&gt;&lt;MESSAGE&gt;&lt;/MESSAGE&gt;&lt;/STATUS&gt;&lt;/QUERY&gt;
  CPF=17834368168;YEARS=2008; &lt;?xml version='1.0' encoding='ISO-8859-1'?&gt;&lt;QUERY&gt;&lt;RESTITUICAO&gt;&lt;CPF&gt;17834368168&lt;/CPF&gt;&lt;ANO&gt;2008&lt;/ANO&gt;&lt;SITUACAODECLARACAO&gt;Sua declaração não consta na base de dados da Receita Federal&lt;/SITUACAODECLARACAO&gt;&lt;DATACONSULTA&gt;05/01/2012&lt;/DATACONSULTA&gt;&lt;/RESTITUICAO&gt;&lt;STATUS&gt;&lt;RESULT&gt;TRUE&lt;/RESULT&gt;&lt;MESSAGE&gt;&lt;/MESSAGE&gt;&lt;/STATUS&gt;&lt;/QUERY&gt;
  CPF=17834368168;YEARS=2007; &lt;?xml version='1.0' encoding='ISO-8859-1'?&gt;&lt;QUERY&gt;&lt;RESTITUICAO&gt;&lt;CPF&gt;17834368168&lt;/CPF&gt;&lt;ANO&gt;2007&lt;/ANO&gt;&lt;SITUACAODECLARACAO&gt;Sua declaração consta como Pedido de Regularização(PR), na base de dados da Secretaria da Receita Federal do Brasil&lt;/SITUACAODECLARACAO&gt;&lt;DATACONSULTA&gt;05/01/2012&lt;/DATACONSULTA&gt;&lt;/RESTITUICAO&gt;&lt;STATUS&gt;&lt;RESULT&gt;TRUE&lt;/RESULT&gt;&lt;MESSAGE&gt;&lt;/MESSAGE&gt;&lt;/STATUS&gt;&lt;/QUERY&gt;
  CPF=17834368168;YEARS=2006; &lt;?xml version='1.0' encoding='ISO-8859-1'?&gt;&lt;QUERY&gt;&lt;RESTITUICAO&gt;&lt;CPF&gt;17834368168&lt;/CPF&gt;&lt;ANO&gt;2006&lt;/ANO&gt;&lt;SITUACAODECLARACAO&gt;Sua declaração não consta na base de dados da Receita Federal&lt;/SITUACAODECLARACAO&gt;&lt;DATACONSULTA&gt;05/01/2012&lt;/DATACONSULTA&gt;&lt;/RESTITUICAO&gt;&lt;STATUS&gt;&lt;RESULT&gt;TRUE&lt;/RESULT&gt;&lt;MESSAGE&gt;&lt;/MESSAGE&gt;&lt;/STATUS&gt;&lt;/QUERY&gt;
  </code></pre>
  
  <p>One line, containing each year information about that document. So basically, the output files are <em>5 times longer than the input files</em>.</p>
  
  <p><strong>Performance Issue:</strong></p>
  
  <p>Each file has 400,000 lines, and I have 133 files to process. </p>
  
  <p>At the moment, here is the flow of my app :</p>
  
  <ol>
  <li>Open a file</li>
  <li>Read a line</li>
  <li>Parse it to the new format</li>
  <li>Write the line to the output file</li>
  <li>Goto 2 until there is no left line</li>
  <li>Goto1 until there is no left file</li>
  </ol>
  
  <p>Each input file is about 700MB, and it is taking forever to read files and write the converted version of them to another one. A file with 400KB takes ~30 seconds to achieve the process.</p>
  
  <p><strong>Extra Information:</strong></p>
  
  <p>My machine runs on a Intel i5 processor, with 8GB RAM.</p>
  
  <p>I am not instantiating tons of object to avoid mem. leaking, and I'm using the <code>using</code> clause on input file opening.</p>
  
  <p>What can I do to make it run faster ?</p>