step cleansingintegratingand transforming data table an overview of techniques to handle missing data technique advantage disadvantage omit the values easy to perform you lose the information from an observation set value to null easy to perform not every modeling technique and/or implementation can handle null values impute static value such as or the mean easy to perform can lead to false estimations from model impute value from an estimated or theoretical distribution does not disturb the model as much harder to execute modeling the value (nondependentdoes not disturb the model too much can lead to too much confidence in the model you don' lose information from the other variables in the observation you make data assumptions can artificially raise dependence among the variables harder to execute you make data assumptions which technique to use at what time is dependent on your particular case iffor instanceyou don' have observations to spareomitting an observation is probably not an option if the variable can be described by stable distributionyou could impute based on this howevermaybe missing value actually means "zero"this can be the case in sales for instanceif no promotion is applied on customer basketthat customer' promo is missingbut most likely it' also no price cut deviations from code book detecting errors in larger data sets against code book or against standardized values can be done with the help of set operations code book is description of your dataa form of metadata it contains things such as the number of variables per observationthe number of observationsand what each encoding within variable means (for instance " equals "negative"" stands for "very positivea code book also tells the type of data you're looking atis it hierarchicalgraphsomething elseyou look at those values that are present in set but not in set these are values that should be corrected it' no coincidence that sets are the data structure that we'll use when we're working in code it' good habit to give your data structures additional thoughtit can save work and improve the performance of your program if you have multiple values to checkit' better to put them from the code book into table and use difference operator to check the discrepancy between both tables this wayyou can profit from the power of database directly more on this in