first steps in big data parts data map(lambda : split(',')firstline parts first(datalines parts filter(lambda : !firstlinedef cleans(row)row [ str(float(row [ ][:- ])/ return [ encode('utf 'replace( " ","lower(for in rowdatalines datalines map(lambda xcleans( )column (index has formatted numbers we don' need that sign executes data cleaning line by line grabs first line splits data set with comma (,delimiter this is the end of line delimiter for this file encodes everything in utf replaces underscores with spacesand lowercases everything cleaning function will use power of spark to clean data the input of this function will be line of data grabs all lines but first linebecause first line is only variable names let' dive little further into the details for each step step starting up spark in interactive mode and loading the context the spark context import isn' required in the pyspark console because context is readily available as variable sc you might have noticed this is also mentioned when opening pysparkcheck out figure in case you overlooked it we then load hive context to enable us to work interactively with hive if you work interactively with sparkthe spark and hive contexts are loaded automaticallybut if you want to use it in batch mode you need to load it manually to submit the code in batch you would use the spark-submit filename py command on the horton sandbox command line from pyspark import sparkcontext from pyspark sql import hivecontext sc sparkcontext(sqlcontext hivecontext(scwith the environment set upwe're ready to start parsing the csv file step reading and parsing the csv file next we read the file from the hadoop file system and split it at every comma we encounter in our code the first line reads the csv file from the hadoop file system the second line splits every line when it encounters comma our csv parser is naive by design because we're learning about sparkbut you can also use the csv package to help you parse line more correctly data sc textfile("//loanstats csv"parts data map(lambda : split(',')