figure retrieve file status on hadoop via the pyspark console with the file ready and waiting for us on hadoopwe can move on to data preparation using sparkbecause it' not clean enough to directly store in hive step data preparation now that we've downloaded the data for analysiswe'll use spark to clean the data before we store it in hive data preparation in spark cleaning data is often an interactive exercisebecause you spot problem and fix the problemand you'll likely do this couple of times before you have clean and crisp data an example of dirty data would be string such as "usa"which is improperly capitalized at this pointwe no longer work in jobs py but use the pyspark command line interface to interact directly with spark spark is well suited for this type of interactive analysis because it doesn' need to save the data after each step and has much better model than hadoop for sharing data between servers ( kind of distributed memorythe transformation consists of four parts start up pyspark (should still be open from section and load the spark and hive context read and parse the csv file split the header line from the data clean the data okayonto business the following listing shows the code implementation in the pyspark console listing connecting to apache spark creates hive context imports spark context --not necessary when working directly in pyspark imports hive context from pyspark import sparkcontext from pyspark sql import hivecontext #sc sparkcontext(sqlcontext hivecontext(scdata sc textfile("//loanstats csv"in the pyspark sessionthe spark context is automatically present in other cases (zeppelin notebookyou'll need to create this explicitly loads in data set from hadoop directory