title: Create a custom sklearn TransformerMixin that transforms categorical variables consistently
body: <p>This question is not a <a href="https://stackoverflow.com/questions/48034035/how-to-consistently-hot-encode-dataframes-with-changing-values">duplicate</a> as someone suggested. Why? Because in that example, all possible values ARE KNOWN. In this example, they aren't. Further, this question - in addition to using a custom converter on unknown values - is asking specifically how to perform the transform in the same exact way as the initial transform. Once again I can tell I'll have to answer my own question eventually.</p>
  
  <hr>
  
  <p>When creating a custom scikit-learn transformer, how can you guarantee or "force" the transform method to output only the columns it was fitted with originally?</p>
  
  <p>Below illustrates. This is my example transformer.</p>
  
  <pre><code>import numpy as np
  import pandas as pd
  from sklearn.base import TransformerMixin
  from sklearn.linear_model import LogisticRegression
  
  class DFTransformer(TransformerMixin):
  
      def fit(self, df, y=None, **fit_params):
          return self
  
      def transform(self, df, **trans_params):
          self.df = df
          self.STACKER = pd.DataFrame()
  
          for col in self.df:
              dtype = self.df[col].dtype.name
              if dtype == 'object':
                  self.STACKER = pd.concat([self.STACKER, self.get_dummies(col)], axis=1)
              elif dtype == 'int64':
                  self.STACKER = pd.concat([self.STACKER, self.cut_it(col)], axis=1)
  
          return self.STACKER
  
      def get_dummies(self, name):
          return pd.get_dummies(self.df[name], prefix=name)
  
      def cut_it(self, name, bins=5):
          s = self.df[name].copy()
          return pd.get_dummies(pd.cut(s, bins), prefix=name)
  </code></pre>
  
  <p>Here's some dummy data. One of my methods uses <code>pd.cut</code> in-efforts to bin large ranges of ints or floats. Another method uses <code>pd.get_dummies</code> which turns unique values into columns.</p>
  
  <pre><code>df = pd.DataFrame({'integers': np.random.randint(2000, 20000, 30, dtype='int64'),
                     'categorical': np.random.choice(list('ABCDEFGHIJKLMNOP'), 30)},
                    columns=['integers', 'categorical'])
  
  trans = DFTransformer()
  X = trans.fit_transform(df)
  y = np.random.binomial(1, 0.5, 30)
  lr = LogisticRegression()
  lr.fit(X, y)
  
  X_test = pd.DataFrame({'integers': np.random.randint(2000, 60000, 30, dtype='int64'),
                     'categorical': np.random.choice(list('ABGIOPXYZ'), 30)},
                    columns=['integers', 'categorical'])
  lr.predict(trans.transform(X_test))
  </code></pre>
  
  <p>The issue I'm having is that when I go and transform the "test" data (the data I'd like to make predictions on), it's highly likely that the conversion won't output the same exact columns due to the different categorical values (ex: obscure values that maybe appear once and then are never seen or heard from again).</p>
  
  <p>For example, the above code produces this error:</p>
  
  <pre><code>Traceback (most recent call last):
    File "C:/Users/myname/Downloads/SO009949884.py", line 44, in &lt;module&gt;
      lr.predict(trans.transform(X_test))
    File "C:\python36\lib\site-packages\sklearn\linear_model\base.py", line 324, in predict
      scores = self.decision_function(X)
    File "C:\python36\lib\site-packages\sklearn\linear_model\base.py", line 305, in decision_function
      % (X.shape[1], n_features))
  ValueError: X has 14 features per sample; expecting 20
  </code></pre>
  
  <p><strong>Question</strong>: how do I go about ensuring my transform method transforms my test data the same way?</p>
  
  <p>One bad solution I can think of is: transform training data, transform testing data, see where columns intersect, modify my transform function to limit output to those columns. Or, fill in blank columns for those that are missing. This is not scalable. Surely there's a better way? I don't want to have to know what the output columns must be before-hand.</p>
  
  <p><strong>My overall goal</strong> is to convert categorical variables in a consistent way across train and test datasets. I have 150+ columns to transform!</p>