title: Efficiently inferring dataframe schema based on key in pyspark
body: <p>I have a large dataset of lines (rows) of json.  The rows have several fields and the fields that are present depend on one of the json fields in that row.  Here is an small example:</p>
  
  <pre><code>%pyspark
  data = sc.parallelize([{'key':'k1','a':1.0,'b':2.0},
                      {'key':'k1','a':1.0,'b':20.0},
                      {'key':'k1','a':100.0,'b':.2},
                      {'key':'k2','y':10.0,'z':20.0},
                      {'key':'k2','y':1.0,'z':250.0},
                      {'key':'k1','a':1.0,'b':2.0},], 2)
  </code></pre>
  
  <p>My goal is to get this data into a Dataframe without having to specify the schema.  Pyspark has (at least) two functions to help with this: 1) <code>toDF()</code>, which just takes the first row of data as the schema and 2) <code>sqlContext.createDataFrame()</code> where you can specify the proportion of rows to sample in order to infer the schema.  e.g.:</p>
  
  <pre><code>data.toDF().show()
  +-----+----+---+
  |    a|   b|key|
  +-----+----+---+
  |  1.0| 2.0| k1|
  |  1.0|20.0| k1|
  |100.0| 0.2| k1|
  | null|null| k2|
  | null|null| k2|
  |  1.0| 2.0| k1|
  +-----+----+---+
  
  sqlContext.createDataFrame(data,samplingRatio=1).show()
  +-----+----+---+----+-----+
  |    a|   b|key|   y|    z|
  +-----+----+---+----+-----+
  |  1.0| 2.0| k1|null| null|
  |  1.0|20.0| k1|null| null|
  |100.0| 0.2| k1|null| null|
  | null|null| k2|10.0| 20.0|
  | null|null| k2| 1.0|250.0|
  |  1.0| 2.0| k1|null| null|
  +-----+----+---+----+-----+
  </code></pre>
  
  <p><code>sqlContext.createDataFrame()</code> does what I want, but since I only have maybe five keys in 4 billion rows, I am thinking that there must be a faster way to infer the schema.  Also, some keys are very rare, so I can't get away with making <code>samplingRatio</code> smaller.</p>
  
  <p>Is there an elegant and fast way to have the schema inferred given there are only a few row types?</p>