title: Recording high frequency data without dropping packets
body: <p>I am using QuickFix with python bindings, along with pandas for data management. </p>
  
  <p>I have been dealing with this issue for awhile, and have not found any clear questions/answers relating to it on SO or Google. It relates to code efficiency and architecture in a low-latency environment. </p>
  
  <p>I am recording financial data. It is extremely high frequency. During fast periods, (small-sized) messages arrive every 15 milliseconds or so. QuickFix passes each message to a message cracker that I wrote, which does the following: </p>
  
  <ol>
  <li>parses the message with <code>re</code></li>
  <li>convert datatype of each element of the message (about 8 of them in this case)</li>
  <li>update the values of a pandas data frame with the 8 elements</li>
  <li>open a <code>.csv</code> file on the local computer, append the line of 8 elements, close the file.</li>
  </ol>
  
  <p>A pretty simple process, but multiply this by several hundred markets, and what happens is my computer is unable to deal. Anywhere between 2 - 100 times per day, the computer chokes, falls offline, and I lose about 20 seconds of data (c. 13,000 samples!) </p>
  
  <p>I am presently looking at PyTables, to see if I can speed up my process. But I do not know enough about computer science to really get to the heart of the speed issue, and would appreciate some wisdom. </p>
  
  <p>Is the problem the <code>.csv</code> file? Can I use PyTables and HDF5 to speed things up? What would be the 'right' way of doing something like this? </p>