title: mean from pandas and numpy differ
body: <p>I have a MEMS IMU on which I've been collecting data and I'm using pandas to get some statistical data from it. There are 6 32-bit floats collected each cycle. Data rates are fixed for a given collection run. The data rates vary between 100Hz and 1000Hz and the collection times run as long as 72 hours. The data is saved in a flat binary file. I read the data this way:</p>
  
  <pre><code>import numpy as np
  import pandas as pd
  dataType=np.dtype([('a','&lt;f4'),('b','&lt;f4'),('c','&lt;f4'),('d','&lt;f4'),('e','&lt;f4'),('e','&lt;f4')])
  df=pd.DataFrame(np.fromfile('FILENAME',dataType))
  df['c'].mean()
  -9.880581855773926
  x=df['c'].values
  x.mean()
  -9.8332081
  </code></pre>
  
  <p>-9.833 is the correct result. I can create a similar result that someone should be able to repeat this way:</p>
  
  <pre><code>import numpy as np
  import pandas as pd
  x=np.random.normal(-9.8,.05,size=900000)
  df=pd.DataFrame(x,dtype='float32',columns=['x'])
  df['x'].mean()
  -9.859579086303711
  x.mean()
  -9.8000648778888628
  </code></pre>
  
  <p>I've repeated this on linux and windows, on AMD and Intel processors, in Python 2.7 and 3.5. I'm stumped. What am I doing wrong?
  And get this:</p>
  
  <pre><code>x=np.random.normal(-9.,.005,size=900000)
  df=pd.DataFrame(x,dtype='float32',columns=['x'])
  df['x'].mean()
  -8.999998092651367
  x.mean()
  -9.0000075889406528
  </code></pre>
  
  <p>I could accept this difference. It's at the limit of the precision of 32 bit floats.</p>
  
  <p>NEVERMIND. I wrote this on Friday and the solution hit me this morning. It is a floating point precision problem exacerbated by the large amount of data. I needed to convert the data into 64 bit float on the creation of the dataframe this way:</p>
  
  <pre><code>df=pd.DataFrame(np.fromfile('FILENAME',dataType),dtype='float64')
  </code></pre>
  
  <p>I'll leave the post should anyone else run into a similar issue.</p>