Data quality improvement as a part of preprocessing: Imputation I have a python pandas dataframe representing a superset. The data contains a lot of nulls which I want to overwrite with real values.

the superset has:


both numerical and categorical data
some nulls for most attributes
multi class attributes (attributes can have multiple values)
It is not time dependent
each row is a unique person


It would be neat to use machine learning to fill in the nulls, any recommendations on how I can do this?

(I guess that I can tranform the categorical data to numerical if required)
Firstly, filling a column which has most values as null might not make sense. You'll have to look at the nature of the business problem you face. You can try dropping those columns. You can drop the rows which contain null values if the number of null values for columns in that row fall below a particular threshold (based on your data, of course). 
Use: dataset.dropna(inplace=True) to achieve this

That being said, there are multiple ways you can fill (impute) null values:


You can try filling values based on the mean (or median or mode): dataset.fillna(dataset.mean(), inplace=True). However this approach has it's limitations if your column is non-numeric.


If you want to use an sklearn implementation, try using the Imputer method.

from sklearn.preprocessing import Imputer

impute = Imputer(missing_values='NaN', strategy='mean', axis=0)

impute.fit(dataset)

dataset = impute.transform(dataset)


For categorical columns, try using fillna with the most frequent value in a column:


dataset[column].fillna(value=df['column1'].value_counts().index[0], inplace =True)

more information on this method can be found here: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.fillna.html


Try using a KNN to fill in missing values. There's a package called fancyimpute which will allow you to do this:


from fancyimpute import KNN

dataset = pd.DataFrame(KNN(number_of_neighbors).complete(dataset))

You'll need to choose the number of neighbors appropriately. Also, fancyimpute needs a numpy array as input. 

There are many other methods, but these should do the trick.