first steps in big data we download the file "loanstats csv zipfrom the lending club' website at resources lendingclub com/loanstats csv zip and unzip it we use methods from the requestszipfileand stringio python packages to respectively download the datacreate virtual fileand unzip it this is only single fileif you want all their data you could create loopbut for demonstration purposes this will do as we mentioned beforean important part of this case study will be data preparation with big data technologies before we can do sohoweverwe need to put it on the hadoop file system pywebhdfs is package that allows you to interact with the hadoop file system from python it translates and passes your commands to rest calls for the webhdfs interface this is useful because you can use your favorite scripting language to automate tasksas shown in the following listing listing storing data on hadoop stores it locally because we need to transfer it to hadoop file system does preliminary data cleaning using pandasremoves top row and bottom rows because they're useless opening original file will show you this import pandas as pd from pywebhdfs webhdfs import pywebhdfsclient subselection_csv pd read_csv(unzipped open('loanstats csv')skiprows= ,skipfooter= ,engine='python'stored_csv subselection_csv to_csv(/stored_csv csv'hdfs pywebhdfsclient(user_name="hdfs",port= ,host="sandbox"hdfs make_dir(''with open(/stored_csv csv'as file_datahdfs create_file('/loanstats csv',file_dataoverwrite=trueconnects to hadoop sandbox creates csv file on hadoop file system creates folder "on hadoop file system opens locally stored csv we had already downloaded and unzipped the file in listing now in listing we made sub-selection of the data using pandas and stored it locally then we created directory on hadoop and transferred the local file to hadoop the downloaded data is in csv format and because it' rather smallwe can use the pandas library to remove the first line and last two lines from the file these contain comments and will only make working with this file cumbersome in hadoop environment the first line of our code imports the pandas packagewhile the second line parses the file into memory and removes the first and last two data lines the third code line saves the data to the local file system for later use and easy inspection before moving onwe can check our file using the following line of codeprint hdfs get_file_dir_status('/loanstats csv'the pyspark console should tell us our file is safe and well on the hadoop systemas shown in figure