title: Impala median calculation on big data
body: <p>I've got access to some data that's hundreds of millions of rows for any given month.  3 features : a string representing a date, a string representing a type and a value representing an amount.  </p>
  
  <p>Having access to python and impala(SQL), what's the best approach to calculate the median on these millions of rows for each type in each month?</p>
  
  <p>If I use a simple group by : the type and substring of the date part to get the month eg substring(date,1,4) and use the APPX_MEDIAN function for the median I just end up running out of memory for the Impala query.</p>
  
  <p>If I try to take the raw data down as a CSV (say with DBeaver), it's huge - GBs in size, too big to fit into the memory of the VM I have access to that would hold the CSV if I try to push it into a python pandas dataframe.</p>
  
  <p>I'm not familiar with patterns that deal with big data, so any tips at all would be very much appreciated.  I'm struggling to perform a simple calculation due to the sheer size of the data.  </p>