title: Python: Fastest way to iterate this through a large file
body: <p>Right, I'm iterating through a large binary file</p>
  
  <p>I need to minimise the time of this loop:</p>
  
  <pre><code>def NB2(self, ID_LEN):
      r1=np.fromfile(ReadFile.fid,dTypes.NB_HDR,1)
      num_receivers=r1[0][0]
      num_channels=r1[0][1]
      num_samples=r1[0][5]
  
      blockReturn = np.zeros((num_samples,num_receivers,num_channels))
  
      for rec in range(0,num_receivers):
          for chl in range(0,num_channels):
              for smpl in range(0,num_samples):
                  r2_iq=np.fromfile(ReadFile.fid,np.int16,2)
                  blockReturn[smpl,rec,chl] = np.sqrt(math.fabs(r2_iq[0])*math.fabs(r2_iq[0]) + math.fabs(r2_iq[1])*math.fabs(r2_iq[1]))
  
      return blockReturn
  </code></pre>
  
  <p>So, what's going on is as follows:
  r1 is the header of the file, dTypes.NB_HDR is a type I made:</p>
  
  <pre><code>NB_HDR= np.dtype([('f3',np.uint32),('f4',np.uint32),('f5',np.uint32),('f6',np.int32),('f7',np.int32),('f8',np.uint32)])
  </code></pre>
  
  <p>That gets all the information about the forthcoming data block, and nicely puts us in the right position within the file (the start of the data block!). </p>
  
  <p>In this data block there is:
  4096 samples per channel,
  4 channels per receiver,
  9 receivers.</p>
  
  <p>So num_receivers, num_channels, num_samples will always be the same (at the moment anyway), but as you can see this is a fairly large amount of data. Each 'sample' is a pair of int16 values that I want to find the magnitude of (hence Pythagoras).</p>
  
  <p>This NB2 code is executed for each 'Block' in the file, for a 12GB file (which is how big they are) there are about 20,900 Blocks, and I've got to iterate through 1000 of these files (so, 12TB overall). Any speed advantage even it's it's milliseconds would be massively appreciated.</p>
  
  <p>EDIT: Actually it might be of help to know how I'm moving around inside the file. I have a function as follows:</p>
  
  <pre><code>def navigateTo(self, blockNum, indexNum):
      ReadFile.fid.seek(ReadFile.fileIndex[blockNum][indexNum],0)
      ReadFile.currentBlock = blockNum
      ReadFile.index = indexNum
  </code></pre>
  
  <p>Before I run all this code I scan the file and make a list of index locations at ReadFile.fileIndex that I browse using this function and then 'seek' to the absolute location - is this efficient?</p>
  
  <p>Cheers</p>