How to impute missing value in Test Set using a custom Imputer created on training dataset
The way you are imputing your feature can't be replicated in the test set, because it needs knowledge of the target classes!
You need to select a different imputation strategy, that doesn't rely on your target feature.

Assuming that you are using another feature, the same way you were using your target, you need to store the value(s) you are imputing each column with in the training set and then impute the test set with the same values as the training set. This would look like this:
# we have two dataframes, train_df and test_df

impute_values = train_df.groupby('Another Feature')['Feature'].mean()

train_df['Feature'] = pd.Series(train_df['Feature'].values, index=train_df['Another Feature']).fillna(impute_values).reset_index(drop=True)

# train your model ...

test_df['Feature'] = pd.Series(test_df['Feature'].values, index=test_df['Another Feature']).fillna(impute_values).reset_index(drop=True)


Example:
train_df = pd.DataFrame({'f1': ['a'] * 5 + ['b'] * 5, 'f2': range(10)})
test_df = pd.DataFrame({'f1': ['a'] * 3 + ['b'] * 7, 'f2': range(10, 20)})
train_df.loc[[1, 6], 'f2'] = np.nan
test_df.loc[[1, 6], 'f2'] = np.nan

impute_values = train_df.groupby('f1')['f2'].mean()

train_df['f2'] = pd.Series(train_df['f2'].values, index=train_df['f1']).fillna(impute_values).reset_index(drop=True)

test_df['f2'] = pd.Series(test_df['f2'].values, index=test_df['f1']).fillna(impute_values).reset_index(drop=True)