title: How to create an external hive table using parquet files consisting of data with complex structs data type
body: <p>I have a set of parquet files which contain data for a table called people. Now this data in the parquet files consist of complex data types like structs etc. the schema of the data in the parquet files has been attached below
  SCHEMA:</p>
  
  <pre><code>|-- distinct_id: string (nullable = true)
   |-- android_app_version: string (nullable = true)
   |-- android_app_version_code: string (nullable = true)
   |-- android_brand: string (nullable = true)
   |-- android_devices: array (nullable = true)
   |    |-- element: string (containsNull = true)
   |-- android_lib_version: string (nullable = true)
   |-- android_manufacturer: string (nullable = true)
   |-- android_os: string (nullable = true)
   |-- android_os_version: string (nullable = true)
   |-- android_push_error: string (nullable = true)
   |-- browser: string (nullable = true)
   |-- browser_version: double (nullable = true)
   |-- campaigns: array (nullable = true)
   |    |-- element: long (containsNull = true)
   |-- country_code: string (nullable = true)
   |-- deliveries: array (nullable = true)
   |    |-- element: long (containsNull = true)
   |-- initial_referrer: string (nullable = true)
   |-- initial_referring_domain: string (nullable = true)
   |-- ios_app_release: string (nullable = true)
   |-- ios_app_version: string (nullable = true)
   |-- ios_device_model: string (nullable = true)
   |-- ios_devices: array (nullable = true)
   |    |-- element: string (containsNull = true)
   |-- ios_lib_version: string (nullable = true)
   |-- ios_version: string (nullable = true)
   |-- last_seen: string (nullable = true)
   |-- notifications: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- $time: string (nullable = true)
   |    |    |-- campaign_id: long (nullable = true)
   |    |    |-- message_id: long (nullable = true)
   |    |    |-- message_subtype: string (nullable = true)
   |    |    |-- message_type: string (nullable = true)
   |    |    |-- time: string (nullable = true)
   |    |    |-- type: string (nullable = true)
   |-- os: string (nullable = true)
   |-- predict_grade: string (nullable = true)
   |-- region: string (nullable = true)
   |-- swift_lib_version: string (nullable = true)
   |-- timezone: string (nullable = true)
   |-- area: string (nullable = true)
   |-- country: string (nullable = true)
   |-- dob: string (nullable = true)
   |-- date: string (nullable = true)
   |-- default_languages: string (nullable = true)
   |-- email: string (nullable = true)
   |-- first_app_launch: string (nullable = true)
   |-- first_app_launch_date: string (nullable = true)
   |-- first_login: boolean (nullable = true)
   |-- gaid: string (nullable = true)
   |-- lr_age: string (nullable = true)
   |-- lr_birthdate: string (nullable = true)
   |-- lr_country: string (nullable = true)
   |-- lr_gender: string (nullable = true)
   |-- language: array (nullable = true)
   |    |-- element: string (containsNull = true)
   |-- languages: string (nullable = true)
   |-- languages_disabled: string (nullable = true)
   |-- languages_selected: string (nullable = true)
   |-- launched: string (nullable = true)
   |-- location: string (nullable = true)
   |-- media_id: string (nullable = true)
   |-- no_of_logins: long (nullable = true)
   |-- pop-strata: string (nullable = true)
   |-- price: string (nullable = true)
   |-- random_number: long (nullable = true)
   |-- second_name: string (nullable = true)
   |-- state: string (nullable = true)
   |-- state_as_per_barc: string (nullable = true)
   |-- total_app_opens: long (nullable = true)
   |-- total_app_sessions: string (nullable = true)
   |-- total_sessions: string (nullable = true)
   |-- town: string (nullable = true)
   |-- user_type: string (nullable = true)
   |-- userid: string (nullable = true)
   |-- appversion: string (nullable = true)
   |-- birthdate: string (nullable = true)
   |-- campaign: string (nullable = true)
   |-- city: string (nullable = true)
   |-- media_source: string (nullable = true)
   |-- last_name: string (nullable = true)
   |-- first_name: string (nullable = true)
   |-- ios_ifa: string (nullable = true)
   |-- android_model: string (nullable = true)
   |-- age: string (nullable = true)
   |-- uid: string (nullable = true)
  </code></pre>
  
  <p>What I want is to eventually create an hive ext table pointing to the data in parquet files. One solution could be flattening or using SQL explode to diversify the structs into individual column data, but I end up getting null values for all the columns which are originally of struct data type. The parquet files are located in a azure blob location.</p>
  
  <p>I tried loading the parquet files in a dataframe in sparksql but it gives null values for columns with complex data type:</p>
  
  <p><img src="https://i.stack.imgur.com/YmvXX.png" alt="enter image description here"></p>