title: Read an unsupported mix of union types from an Avro file in Apache Spark
body: <p>I'm trying to switch from reading csv flat files to avro files on spark.
  following <a href="https://github.com/databricks/spark-avro" rel="noreferrer">https://github.com/databricks/spark-avro</a>
  I use:</p>
  
  <pre><code>import com.databricks.spark.avro._
  val sqlContext = new org.apache.spark.sql.SQLContext(sc)
  val df = sqlContext.read.avro("gs://logs.xyz.com/raw/2016/04/20/div1/div2/2016-04-20-08-28-35.UTC.blah-blah.avro")
  </code></pre>
  
  <p>and get</p>
  
  <pre><code>java.lang.UnsupportedOperationException: This mix of union types is not supported (see README): ArrayBuffer(STRING)
  </code></pre>
  
  <p>the readme file states clearly:</p>
  
  <blockquote>
    <p>This library supports reading all Avro types, with the <strong>exception of
    complex union types</strong>. It uses the following mapping from Avro types to
    Spark SQL types:</p>
  </blockquote>
  
  <p>when i try to textread the same file I can see the schema</p>
  
  <pre><code>val df = sc.textFile("gs://logs.xyz.com/raw/2016/04/20/div1/div2/2016-04-20-08-28-35.UTC.blah-blah.avro")
  df.take(2).foreach(println)
  </code></pre>
  
  <blockquote>
    <p>{"name":"log_record","type":"record","fields":[{"name":"request","type":{"type":"record","name":"request_data","fields":[{"name":"datetime","type":"string"},{"name":"ip","type":"string"},{"name":"host","type":"string"},{"name":"uri","type":"string"},{"name":"request_uri","type":"string"},{"name":"referer","type":"string"},{"name":"useragent","type":"string"}]}}</p>
    
    <p>&lt;------- an excerpt of the full reply -------></p>
  </blockquote>
  
  <p>since I have little control on the format I'm getting these files in,
  my question here is - <strong>is there a workaround someone tested and can recommend?</strong></p>
  
  <p>I use gc dataproc with </p>
  
  <blockquote>
    <p>MASTER=yarn-cluster spark-shell --num-executors 4 --executor-memory 4G  --executor-cores 4 --packages com.databricks:spark-avro_2.10:2.0.1,com.databricks:spark-csv_2.11:1.3.0</p>
  </blockquote>
  
  <p>any help would be greatly appreciated.....</p>