title: How to "reduce" multiple json tables stored in a column of an RDD to a single RDD table as efficiently as possible
body: <p>Does concurrent access to append rows using union  in a dataframe using following code will work correctly? Currently showing type error</p>
  
  <pre><code>from pyspark.sql.types import *
  schema = StructType([
      StructField("owreg", StringType(), True),StructField("we", StringType(), True)
          ,StructField("aa", StringType(), True)
          ,StructField("cc", StringType(), True)
          ,StructField("ss", StringType(), True)
          ,StructField("ss", StringType(), True)
          ,StructField("sss", StringType(), True)
  ])
  
  f = sqlContext.createDataFrame(sc.emptyRDD(), schema)
  def dump(l,jsid):
      if not l.startswith("&lt;!E!&gt;"):
           f=f.unionAll(sqlContext.read.json(l))
  savedlabels.limit(10).foreach(lambda a: dump(a.labels,a.job_seq_id))
  </code></pre>
  
  <p>Assume that sqlContext.read.json(l) will read a json and output a RDD with the same schema</p>
  
  <p>The pattern is that I want to "reduce" multiple json tables stored in a column of an RDD to an RDD table as efficiently as possible.</p>
  
  <pre><code>def dump(l,jsid):
      if not l.startswith("&lt;!E!&gt;"):
          f=f.unionAll(sc.parallelize(json.loads(l)).toDF())
  </code></pre>
  
  <p>The above code will also no work as sc.parallelize is being invoked by the worker threads. Hence how to solve this problem?</p>