title: How do I fill missing categorical variables on a time based axis, after a groupby on a pandas dataframe? (ie: the missing values fillna won't catch)
body: <p><strong>TL,DR</strong>:</p>
  <p>I have a dataset showing total covid vaccinations. I have grouped this data by country and date. However some countries have no entries (not Na or NaN, just non-existent records) for certain dates.</p>
  <p>I want each country to have a value for total vaccinations for the whole date range of my dataset, naturally filling it with 0 for any dates that currently have no data.</p>
  <p>What is the easiest way to achieve this? I'm guessing this is a pretty trivial question, but I'm just starting with python and pandas.</p>
  <hr />
  <p><strong>In more detail:</strong></p>
  <p>I'm using the following dataset: <a href="https://www.kaggle.com/gpreda/covid-world-vaccination-progress" rel="nofollow noreferrer">https://www.kaggle.com/gpreda/covid-world-vaccination-progress</a>.
  I add it to a pandas dataframe:</p>
  <pre><code>covid_df = pd.read_csv('country_vaccinations.csv', parse_dates=True)
  </code></pre>
  <p>I'm grouping the dataset by country and date and summing the total vaccinations:</p>
  <pre><code>    grouped_country_df = covid_df[['country','total_vaccinations','date']].
  groupby(['country','date'], as_index = False).sum('total_vaccinations')
  </code></pre>
  <p>However, I get left with some countries that simply don't show up for certain dates - they aren't NaN or NA, they just aren't present.</p>
  <p>For example:</p>
  <pre><code>print(len(grouped_country_df['date'][grouped_country_df['country'] == 'United States']))
  print(len(grouped_country_df['date'][grouped_country_df['country'] == 'China']))
  print(len(grouped_country_df['date'][grouped_country_df['country'] == 'United Kingdom']))
  </code></pre>
  <p>Returns:</p>
  <pre><code>51
  51
  57
  </code></pre>
  <p>I think this is because the UK started vaccinating earlier than all the others, so there simply aren't any records for China and the US at the start of the vaccination process. So for later dates, they show up as &quot;NaN&quot;, which is easy to fill with 0s using fillna(), but for earlier dates I'm not sure what to do.</p>
  <p>Here's an example for the United States from the original covid_df dataframe to further illustrate my point:</p>
  <pre><code>covid_df[['country','total_vaccinations','date']][covid_df['country'] == 'United States']
  
  
  
  country total_vaccinations  date
  2447    United States   556208.0    2020-12-20
  2448    United States   614117.0    2020-12-21
  2449    United States   NaN 2020-12-22
  2450    United States   1008025.0   2020-12-23
  2451    United States   NaN 2020-12-24
  2452    United States   NaN 2020-12-25
  2453    United States   1944585.0   2020-12-26
  2454    United States   NaN 2020-12-27
  2455    United States   2127143.0   2020-12-28
  </code></pre>
  <p>As you can see, the United States <strong>doesn't have an entry</strong> for dates earlier than the '2020-12-20', so there's nothing for fillna() to fill.</p>
  <p>I came up with the following solution which works, but I'm pretty sure it's very inelegant, and I bet there's an incredibly simply solve staring me in the face:</p>
  <pre><code>new_country_df = pd.DataFrame()
  for country in grouped_covid_countries_df['country'].values:
      dates_df = grouped_covid_countries_df['date'].to_frame().drop_duplicates()
      country_df = grouped_covid_countries_df[['date','total_vaccinations','country']][grouped_covid_countries_df['country'] == country].drop_duplicates()
      temp_df = dates_df.merge(country_df, on='date', how='left').fillna(0)
      temp_df['country'] = country
      new_country_df = pd.concat([new_country_df, temp_df])
  new_country_df['date'][new_country_df['country'] == 'United States']
  </code></pre>
  <p>The reason I need to achieve this is because I want to draw a stacked bar chart plotting the total vaccinations over time, but I get errors if each country value doesn't have an entry for each date, when trying to compute the &quot;bottom&quot; argument of the matplotlib.pyplot.bar() method:</p>
  <pre><code>bottom = len(new_country_df['date'].value_counts()) * [0]
  plt.figure(figsize=(10,5))
  for country in new_country_df['country'].unique():
      x = new_country_df['date'][new_country_df['country'] == country]
      y = new_country_df['total_vaccinations'][new_country_df['country'] == country]
      plt.gca().xaxis.set_major_formatter(dates.ConciseDateFormatter(dates.AutoDateLocator()))
      plt.bar(x,y, label=country, bottom=bottom)
      bottom = bottom + new_country_df['total_vaccinations'][new_country_df['country'] == country]
  plt.gca().legend(loc = 'upper left')
  </code></pre>
  <p><a href="https://i.stack.imgur.com/4ffmF.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4ffmF.png" alt="enter image description here" /></a></p>
  <p>Thanks for reading! I hope my question was clear, please feel free to provide feedback on ways I could improve it or rephrase it.</p>