notice how similar this is to functional programming approach for those who've never encountered ityou can naively read lambda : split(','as "for every input ( row in this case)split this input when it encounters comma as in this case"for every inputmeans "for every row,but you can also read it as "split every row by comma this functional-like syntax is one of my favorite characteristics of spark step split the header line from the data to separate the header from the datawe read in the first line and retain every line that' not similar to the header linefirstline parts first(datalines parts filter(lambda : !firstlinefollowing the best practices in big datawe wouldn' have to do this step because the first line would already be stored in separate file in realitycsv files do often contain header line and you'll need to perform similar operation before you can start cleaning the data step clean the data in this step we perform basic cleaning to enhance the data quality this allows us to build better report after the second stepour data consists of arrays we'll treat every input for lambda function as an array now and return an array to ease this taskwe build helper function that cleans our cleaning consists of reformatting an input such as " , %to and encoding every string as utf- as well as replacing underscores with spaces and lowercasing all the strings the second line of code calls our helper function for every line of the array def cleans(row)row [ str(float(row [ ][:- ])/ return [ encode('utf 'replace( " ","lower(for in rowdatalines datalines map(lambda xcleans( )our data is now prepared for the reportso we need to make it available for our reporting tools hive is well suited for thisbecause many reporting tools can connect to it let' look at how to accomplish this save the data in hive to store data in hive we need to complete two steps create and register metadata execute sql statements to save data in hive in this sectionwe'll once again execute the next piece of code in our beloved pyspark shellas shown in the following listing