title: Any efficient way to read datas from large binary file?
body: <p>I need to handle tens of Gigabytes data in one binary file. Each record in the data file is variable length.</p>
  
  <p>So the file is like:</p>
  
  <pre><code>&lt;len1&gt;&lt;data1&gt;&lt;len2&gt;&lt;data2&gt;..........&lt;lenN&gt;&lt;dataN&gt;
  </code></pre>
  
  <p>The data contains integer, pointer, double value and so on. </p>
  
  <p>I found python can not even handle this situation. There is no problem if I read the whole file in memory. It's fast. But it seems the <code>struct</code> package is not good at performance. It almost stuck on unpack the bytes.</p>
  
  <p>Any help is appreciated.</p>
  
  <p>Thanks.</p>