handling duplicates another issue with many datasets is the presence of duplicates while data is important and more the merrierduplicates do not add much value per se even moreduplicates help us identify potential areas of errors in recording/collecting the data itself to identify duplicateswe have utility called duplicated(that can applied on the whole dataframe as well as on subset of it we may handle duplicates by fixing the errors and use the duplicated(functionalthough we may also choose to drop the duplicate data points altogether to drop duplicateswe use the method drop_duplicates(the following snippet showcases both functions discussed here df_dropped[df_dropped duplicated(subset=['serial_no'])df_dropped drop_duplicates(subset=['serial_no'],inplace=truethe output of identifying subset of dataframe having duplicate values for the field serial_no is depicted in figure - the second line in the previous snippet simply drops those duplicates figure - dataframe with duplicate serial_no values handling categorical data as discussed in the section "data description,categorical attributes consist of data that can take limited number of values (not always thoughhere in our datasetthe attribute user_type is categorical variable that can take only limited number of values from the allowed set { , , ,dthe algorithms that we would be learning and utilizing in the coming mostly work with numerical data and categorical variables may pose some issues with pandaswe can handle categorical variables in couple of different ways the first one is using the map(functionwhere we simply map each value from the allowed set to numeric value though this may be usefulthis approach should be handled with care and caveats for instancestatistical operations like additionmeanand so onthough syntactically validshould be avoided for obvious reasons (more on this in coming the second method is to convert the categorical variable into indicator variables using the get_dummies(function the function is simply wrapper to generate one hot encoding for the variable in consideration one hot encoding and other encodings can be handled using libraries like sklearn as well (we will see more examples in coming the following snippet showcases both the methods discussed previously using map(and get_dummies(using map to dummy encode type_map={' ': ,' ': ,' ': ,' ': ,np nan:- df['encoded_user_type'df user_type map(type_mapprint(df head()using get_dummies to one hot encode print(pd get_dummies(df,columns=['user_type']head()