title: How to process JSON field from relation database with PYSPARK?
body: <p>I am trying to process JSON column from PostgreSQL database.
  I am able to connect to database using:</p>
  
  <pre><code>import os
  import pyspark
  import findspark
  from pyspark import SparkContext
  from pyspark.sql import SQLContext
  findspark.init(os.environ['SPARK_HOME'])
  
  # DB credentials
  user = os.environ['EVENTS_DEV_UID']
  password = os.environ['EVENTS_DEV_PWD']
  host = os.environ['EVENTS_DEV_HOST']
  port = os.environ['EVENTS_DEV_PORT']
  db = os.environ['EVENTS_DEV_DBNAME']
  
  # Initiate spark session
  sc = SparkContext()
  spark = SQLContext(sc)
  
  # Set properties
  properties = {"user": user, "password": password, "driver": "org.postgresql.Driver"}
  
  # Load data
  df = spark.read.jdbc(
      url = 'jdbc:postgresql://' + host + ':' + port + '/' + db,
      table = 'events',
      properties = properties)
  </code></pre>
  
  <p>The problem starts with casting JSON field. Spark doesn't recognize struct format of <code>params</code>. When I print schema:</p>
  
  <pre><code>df.printSchema()
  </code></pre>
  
  <p>root <br>
     |-- time: timestamp (nullable = true) <br>
     |-- name: string (nullable = true) <br>
     |-- params: string (nullable = true) <br></p>
  
  <p>When I try to cast string to struct:</p>
  
  <pre><code>df = df.withColumn('params', df.params.cast('struct'))
  </code></pre>
  
  <p>I am getting following error:</p>
  
  <pre><code>ParseException: '\nDataType struct is not supported.(line 1, pos 0)\n\n== SQL ==\nstruct\n^^^\n'
  </code></pre>
  
  <p>I guess problem is the escape characters. Anybody has idea how to proceed?</p>