Pandas Groupby operation on a large dataset
Consider the following points:


Pandas has a good datetime functionality; you shouldn't cast into strings and then later on group by those. It's unnatural and slow. Instead, just do:

isd['YYYYMM'] = pd.to_datetime(isd['YYYYMMDDHH'],format='%Y%m%d%H')


And then in the groupby, you can simply do

stn_month = isd.groupby(['STATION', isd['YYYYMM'].dt.to_period('M')]) ...

In general, using apply is usually not great for performance. First, notice that you are doing a lot of things inside Nstat that are not necessary: all the lines like d['NDU6']  = (df.loc[ df['RH']&lt;=90,'MW']==6 ).sum() are unnecessary in a sense that you can just precompute this outside of the function. As a side note, the way that you write is unnatural to me and I would more simply do:

df[(df['RH'] &lt;= 90) &amp; (df['MW'] == 6)]


Second, the agg function also takes a dictionary so that you can just do:

isd.groupby(['STATION', isd['YYYYMM'].dt.to_period('M')]).agg({'MW' : 'count', 'PW' : 'count', 'Vi': 'mean'})


I hope this will get you started.