title: IndexError when replacing missing values with mode using groupby in pandas
body: <p>I have a dataset which requires missing value treatment.</p>
  
  <pre><code> Column                      Missing Values
  
   Complaint_ID                    0         
   Date_received                   0         
   Transaction_Type                0         
   Complaint_reason                0         
   Company_response              22506         
   Date_sent_to_company            0         
   Complaint_Status                0         
   Consumer_disputes             7698
  </code></pre>
  
  <p>Now the problem is, when I try to replace the missing <code>values</code> with mode of other <code>columns</code> using <code>groupby</code>:</p>
  
  <p>Code:</p>
  
  <pre><code>data11["Company_response"] = 
  data11.groupby("Complaint_reason").transform(lambda x: x.fillna(x.mode() 
  [0]))["Company_response"]
  
  data11["Consumer_disputes"] = 
  data11.groupby("Transaction_Type").transform(lambda x: x.fillna(x.mode() 
  [0]))["Consumer_disputes"]
  </code></pre>
  
  <p>I get the following error:</p>
  
  <h2>Stacktrace</h2>
  
  <pre><code>Traceback (most recent call last):
  
  File "&lt;ipython-input-89-8de6a010a299&gt;", line 1, in &lt;module&gt;
      data11["Company_response"] = data11.groupby("Complaint_reason").transform(lambda x: x.fillna(x.mode()[0]))["Company_response"]
  
    File "C:\Anaconda3\lib\site-packages\pandas\core\groupby.py", line 3741, in transform
      return self._transform_general(func, *args, **kwargs)
  
    File "C:\Anaconda3\lib\site-packages\pandas\core\groupby.py", line 3699, in _transform_general
      res = path(group)
  
    File "C:\Anaconda3\lib\site-packages\pandas\core\groupby.py", line 3783, in &lt;lambda&gt;
      lambda x: func(x, *args, **kwargs), axis=self.axis)
  
    File "C:\Anaconda3\lib\site-packages\pandas\core\frame.py", line 4360, in apply
      ignore_failures=ignore_failures)
  
    File "C:\Anaconda3\lib\site-packages\pandas\core\frame.py", line 4456, in _apply_standard
      results[i] = func(v)
  
    File "C:\Anaconda3\lib\site-packages\pandas\core\groupby.py", line 3783, in &lt;lambda&gt;
      lambda x: func(x, *args, **kwargs), axis=self.axis)
  
    File "&lt;ipython-input-89-8de6a010a299&gt;", line 1, in &lt;lambda&gt;
      data11["Company_response"] = data11.groupby("Complaint_reason").transform(lambda x: x.fillna(x.mode()[0]))["Company_response"]
  
    File "C:\Anaconda3\lib\site-packages\pandas\core\series.py", line 601, in __getitem__
      result = self.index.get_value(self, key)
  
    File "C:\Anaconda3\lib\site-packages\pandas\core\indexes\base.py", line 2434, in get_value
      return libts.get_value_box(s, key)
  
    File "pandas\_libs\tslib.pyx", line 923, in pandas._libs.tslib.get_value_box (pandas\_libs\tslib.c:18843)
  
    File "pandas\_libs\tslib.pyx", line 939, in pandas._libs.tslib.get_value_box (pandas\_libs\tslib.c:18560)
  
  IndexError: ('index out of bounds', 'occurred at index Consumer_disputes')
  </code></pre>
  
  <p>I have checked the <code>length</code> of the <code>dataframe</code>and all of its columns and it is same: 43266.</p>
  
  <p>I have also found a question similar to this but does not have correct answer: <a href="https://stackoverflow.com/questions/42404776/pandas-fill-in-missing-values-by-mode-having-index-out-of-bounds-error">Click here</a></p>
  
  <p>Please help resolve the error.</p>
  
  <blockquote>
    <p>IndexError: ('index out of bounds', 'occurred at index Consumer_disputes')</p>
  </blockquote>
  
  <p>Here is a snapshot of the dataset if it helps in any way: <a href="https://i.stack.imgur.com/v7xIp.png" rel="nofollow noreferrer">Dataset Snapshot</a></p>
  
  <p>I am using the below code successfully. But it does not serve my purpose exactly. Helps to fill the missing values though.</p>
  
  <pre><code>data11['Company_response'].fillna(data11['Company_response'].mode()[0], 
  inplace=True)
  data11['Consumer_disputes'].fillna(data11['Consumer_disputes'].mode()[0], 
  inplace=True)
  </code></pre>
  
  <p>Edit1: (Attaching Sample)</p>
  
  <p>Input Given:
  <a href="https://i.stack.imgur.com/ygyDx.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ygyDx.png" alt="InputImage"></a></p>
  
  <p>Expected Output:
  <a href="https://i.stack.imgur.com/9StKv.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/9StKv.png" alt="OutputImage"></a></p>
  
  <p>You can see that the missing values for company-response of Tr-1 and Tr-3 are filled by taking mode of Complaint-Reason.
  And similarly for the Consumer-Disputes by taking mode of transaction-type, for Tr-5.</p>
  
  <p>The below snippet consists of the dataframe and the code for those who want to replicate and give it a try.</p>
  
  <h2>Replication Code</h2>
  
  <pre><code>import pandas as pd
  import numpy as np
  
  data11=pd.DataFrame({'Complaint_ID':['Tr-1','Tr-2','Tr-3','Tr-4','Tr-5','Tr-6'],
                      'Transaction_Type':['Mortgage','Credit card','Bank account or service','Debt collection','Credit card','Mortgage'],
                      'Complaint_reason':['Loan servicing, payments, escrow account','Incorrect information on credit report',"Cont'd attempts collect debt not owed","Cont'd attempts collect debt not owed",'Payoff process','Loan servicing, payments, escrow account'],
                      'Company_response':[np.nan,'Company chooses not to provide a public response',np.nan,'Company believes it acted appropriately as authorized by contract or law','Company has responded to the consumer and the CFPB and chooses not to provide a public response','Company disputes the facts presented in the complaint'],
                      'Consumer_disputes':['Yes','No','No','No',np.nan,'Yes']})
  
  data11.isnull().sum()
  
  data11["Company_response"] = data11.groupby("Complaint_reason").transform(lambda x: x.fillna(x.mode()[0]))["Company_response"]
  data11["Consumer_disputes"] = data11.groupby("Transaction_Type").transform(lambda x: x.fillna(x.mode()[0]))["Consumer_disputes"]    
  </code></pre>