transform new dataset for prediction in Python
Question: I train model (for ample _linear_model.LinearRegression_) with some iteration
like `*pd.get_dummies*`

and I get new structure of data Now I take a new dataset & want to predict. I
cann't use _`predict`_ because structures are different. `*pd.get_dummies*`for
new data will give us another number of columns

How can I transform this dataset? By appendind to the previous dataset and
train again with the new data? Or may i use "transform" for new data?

    
    
     import pandas as pd 
    import numpy as np
    from sklearn import linear_model
    df1 = pd.DataFrame({ 'y' : np.array([1., 1., 2., 3., 1.] ,dtype='int32'),
       ....:                      'X' : np.array(["1","1","2","2", "1"])})
    y = df1[df1.columns[0]]
    X = pd.get_dummies(df1['X'])
    lr = linear_model.LinearRegression()
    lr = lr.fit(X, y)
    lr.predict(X)
    

Now i have

    
    
    df2 = pd.DataFrame({ 'y' : 'nan',
       ....:                      'X' : np.array(["3"])})
    Xnew = pd.get_dummies(df2['X'])
    lr.predict(Xnew)
    ValueError: shapes (1,1) and (2,) not aligned: 1 (dim 1) != 2 (dim 0)
    


Answer: I see just this way

    
    
    import numpy as np
    import pandas as pd
    from sklearn import linear_model, metrics, pipeline, preprocessing
    df = pd.DataFrame({'a':range(12), 'b':[1,2,3,1,2,3,1,2,3,3,1,2], 'c':['a', 'b', 'c']*4, 'd': ['m', 'f']*6})
    y = df.a
    num = df[['b']]
    cat = df[['c', 'd']]
    from sklearn.feature_extraction import DictVectorizer
    enc = DictVectorizer(sparse = False)
    enc_data = enc.fit_transform(cat .T.to_dict().values())
    crat = pd.DataFrame(enc_data,  columns=enc.get_feature_names())
    X = pd.concat([crat, num], axis=1)
    cat_columns = ['c=a', 'c=b', 'c=c', 'd=f', 'd=m'] 
    cat_indices = np.array([(column in cat_columns) for column in X.columns], dtype = bool)
    numeric_col = ['b']
    num_indices = np.array([(column in numeric_col) for column in X.columns], dtype = bool)
    reg = linear_model.SGDRegressor()
    estimator = pipeline.Pipeline(steps = [       
        ('feature_processing', pipeline.FeatureUnion(transformer_list = [        
                ('categorical', preprocessing.FunctionTransformer(lambda data: data[:, cat_indices])), 
    
                #numeric
                ('numeric', pipeline.Pipeline(steps = [
                    ('select', preprocessing.FunctionTransformer(lambda data: data[:, num_indices])),
                    ('scale', preprocessing.StandardScaler())            
                            ]))
            ])),
        ('model', reg)
        ]
    )
    estimator.fit(X, y)
    

and now we work witn a new dataset

    
    
    test = pd.DataFrame({ 'b':[1], 'c':['a'], 'd': ['f']})
    cat = test[['c', 'd']]
    num = test[['b']]
    enc_data = enc.transform(cat.T.to_dict().values())
    crat = pd.DataFrame(enc_data,  columns=enc.get_feature_names())
    test = pd.concat([crat, num], axis=1)
    estimator.predict(test)
    

