first steps in big data listing storing data in hive (fullcreates metadatathe spark sql structfield function represents field in structtype the structfield object is comprised of three fieldsname ( string)datatype ( datatype)and "nullable( booleanthe field of name is the name of structfield the field of datatype specifies the data type of structfield the field of nullable specifies if values of structfield can contain none values imports sql data types from pyspark sql types import fields [structfield(field_name,stringtype(),truefor field_name in firstlineschema structtype(fieldsschemaloans sqlcontext createdataframe(datalinesschemacreates data schemaloans registertemptable("loans"frame from data structtype function creates the data schema structtype object requires list of structfields as input registers it as table called loans (datalinesand data schema (schemasqlcontext sql("drop table if exists loansbytitle"sql '''create table loansbytitle stored as parquet as select titlecount( as number from loans group by title order by number desc''sqlcontext sql(sqlsqlcontext sql('drop table if exists raw'sql '''create table raw stored as parquet as select titleemp_title,grade,home_ownership,int_rate,recoveriescollection_recovery_fee,loan_amnt,term from loans''drops table (in case it already existsand stores subset of raw data in hive drops table (in case it already exists)summarizesand stores it in hive loansbytitle represents the sum of loans by job title let' drill deeper into each step for bit more clarification step create and register metadata many people prefer to use sql when they work with data this is also possible with spark you can even read and store data in hive directly as we'll do before you can do thathoweveryou'll need to create metadata that contains column name and column type for every column the first line of code is the imports the second line parses the field name and the field type and specifies if field is mandatory the structtype represents rows as an array of structfields then you place it in dataframe that' registered as (temporarytable in hive