title: pandas: DataFrame.mean() very slow. How can I calculate means of columns faster?
body: <p>I have a rather large CSV file, it contains 9917530 rows (without the header), and 54 columns. Columns are real or integer, only one contains dates. There is a few NULL values on the file, which are translated to <code>nan</code> after I load it to pandas <code>DataFrame</code>, which I do like this:</p>
  
  <pre><code>import pandas as pd
  data = pd.read_csv('data.csv')
  </code></pre>
  
  <p>After loading, which I think was very fast, cause it took around 30 seconds (pretty much the same time as counting lines with the Unix tool <code>wc</code>), the process was taking around 4Gb of RAM (the size of of the file on disk: 2.2 Gb. So far so good.</p>
  
  <p>Then I tried to do the following:</p>
  
  <pre><code>column_means = data.mean()
  </code></pre>
  
  <p>The process' occupied memory grew to ~22Gb very quickly. I could also see the processor (one core) was very very busy - for like three hours, after that I killed the process, cause I needed to use the machine for other things. I have a pretty fast PC with Linux - it has 2 processors, each having 4 cores, so it's 8 cores all together, and 32 Gb of RAM. I cannot believe calculating column means should take so long.</p>
  
  <p>Can anybody explain why <code>DataFrame.mean()</code> is so slow? And more importantly, what is a better way of calculating means of columns of a file like that? Did I not load the file the best way possible, should I use a different function instead of <code>DataFrame.mean()</code> or perhaps a completely different tool?</p>
  
  <p>Many thanks in advance.</p>
  
  <p>EDIT. Here is what <code>df.info()</code> shows:</p>
  
  <pre><code>&lt;class 'pandas.core.frame.DataFrame'&gt;
  Int64Index: 9917530 entries, 0 to 9917529
  Data columns (total 54 columns):
  srch_id                        9917530  non-null values
  date_time                      9917530  non-null values
  site_id                        9917530  non-null values
  visitor_location_country_id    9917530  non-null values
  visitor_hist_starrating        505297  non-null values
  visitor_hist_adr_usd           507612  non-null values
  prop_country_id                9917530  non-null values
  prop_id                        9917530  non-null values
  prop_starrating                9917530  non-null values
  prop_review_score              9902900  non-null values
  prop_brand_bool                9917530  non-null values
  prop_location_score1           9917530  non-null values
  prop_location_score2           7739150  non-null values
  prop_log_historical_price      9917530  non-null values
  position                       9917530  non-null values
  price_usd                      9917530  non-null values
  promotion_flag                 9917530  non-null values
  srch_destination_id            9917530  non-null values
  srch_length_of_stay            9917530  non-null values
  srch_booking_window            9917530  non-null values
  srch_adults_count              9917530  non-null values
  srch_children_count            9917530  non-null values
  srch_room_count                9917530  non-null values
  srch_saturday_night_bool       9917530  non-null values
  srch_query_affinity_score      635564  non-null values
  orig_destination_distance      6701069  non-null values
  random_bool                    9917530  non-null values
  comp1_rate                     235806  non-null values
  comp1_inv                      254433  non-null values
  comp1_rate_percent_diff        184907  non-null values
  comp2_rate                     4040633  non-null values
  comp2_inv                      4251538  non-null values
  comp2_rate_percent_diff        1109847  non-null values
  comp3_rate                     3059273  non-null values
  comp3_inv                      3292221  non-null values
  comp3_rate_percent_diff        944007  non-null values
  comp4_rate                     620099  non-null values
  comp4_inv                      692471  non-null values
  comp4_rate_percent_diff        264213  non-null values
  comp5_rate                     4444294  non-null values
  comp5_inv                      4720833  non-null values
  comp5_rate_percent_diff        1681006  non-null values
  comp6_rate                     482487  non-null values
  comp6_inv                      524145  non-null values
  comp6_rate_percent_diff        193312  non-null values
  comp7_rate                     631077  non-null values
  comp7_inv                      713175  non-null values
  comp7_rate_percent_diff        277838  non-null values
  comp8_rate                     3819043  non-null values
  comp8_inv                      3960388  non-null values
  comp8_rate_percent_diff        1225707  non-null values
  click_bool                     9917530  non-null values
  gross_bookings_usd             276592  non-null values
  booking_bool                   9917530  non-null values
  dtypes: float64(34), int64(19), object(1)None
  </code></pre>