title: Parsing a massive .dat file in Python and doing an arithmetic operation
body: <p>I'm new to Python and relatively new to programming in general.  My basic problem is to parse through a large data file (several million rows of data), stored in a dat file.</p>
  
  <p>Sample data from the file is:</p>
  
  <pre><code>820401001 825029767710821718 8 5 510-180090000 8   9   4
  820401001 8083 7970200661367 7 8 0 3-170090070 0  24   1
  820401001 8082 4745200341-18 4 9 0 3 240080044 0 -20   2
  820401001 8062 5805200461367 2 9 0 3 120066725 0  -7   2
  820401001 8037 5292200491-17 7 7 0 3-170090070 0 -16   2
  </code></pre>
  
  <p>I know the following information:</p>
  
  <ul>
  <li>The length of each row is always 56 characters.  All characters are either a digit or the "-" sign.  Essentially, the data is numerical. </li>
  <li>There are 20 columns to each row, and the columns widths (ie num of characters) are 8, 1, 5, 5, 5, 1, 2, 1, 2, 2, 2, 2, 2, 3, 3, 1, 1, 2, 4, 4</li>
  </ul>
  
  <p>In addition to parsing each line, I need to perform an arithmetic operation on columns 3 and 4.  Specifically, I want to take the 5 characters and divide by 100, eg instead of 29767 I want 297.67</p>
  
  <p>The objective is to create a massive matrix with the resulting values.  Ideally, I'd like to save the matrix in a new file, but I'm unsure how to do that.</p>
  
  <p>The desired output would be something like:</p>
  
  <pre><code>82040100 1 82.50 297.67 71082 1 71 8 8 5 5 10 -1 800 900 0 0 8 9 4
  82040100 1 80.83  79.70 20066 1 36 7 7 8 0  3 -1 700 900 7 0 0 24 1
  </code></pre>
  
  <p>I am aware that I can use the struct library (see code attempt below) but I obtain the error "unpack_from requires a buffer of at least 224 bytes".  I have absolutely no idea what that means.</p>
  
  <p>In addition, I don't know how to perform the operation on columns 3 and 4 in an efficient manner, ie can I do it at the same time when parsing, or do add a separate "if-else" statement?</p>
  
  <pre><code>import struct
  
  fieldwidths = (8, 1, 5, 5, 5, 1, 2, 1, 2, 2, 2, 2, 2, 3, 3, 1, 1, 2, 4, 4) 
  fmtstring = ' '.join('{}{}'.format(abs(fw), 'i') for fw in fieldwidths)
  fieldstruct = struct.Struct(fmtstring)
  
  parse = fieldstruct.unpack_from
  print('fmtstring: {!r}, recsize: {} chars'.format(fmtstring, fieldstruct.size))
  
  print("Opening the file.")
  data_file = open("APR82L.dat", "r")
  
  print("\nReading one line at a time")
  #set to 10 just to test
  for i in range(10):
      line = data_file.readline()
      print(line)
      fields = parse(line)
      print('fields: {}'.format(fields))
  
  #Close the data file
  print("\nClosing the data file")
  data_file.close
  </code></pre>