title: How to convert multiple parquet files into TFrecord files using SPARK?
body: <p>I would like to produce stratified TFrecord files from a large DataFrame based on a certain condition, for which I use <code>write.partitionBy()</code>. I'm also using the tensorflow-connector in SPARK, but this apparently does not work together with a <code>write.partitionBy()</code> operation. Therefore, I have not found another way than to try to work in two steps:</p>
  
  <ol>
  <li>Repartion the dataframe according to my condition, using <code>partitionBy()</code> and write the resulting partitions to parquet files.</li>
  <li>Read those parquet files to convert them into TFrecord files with the tensorflow-connector plugin.</li>
  </ol>
  
  <p>It is the second step that I'm unable to do efficiently. My idea was to read in the individual parquet files on the executors and immediately write them into TFrecord files. But this needs access to the SQLContext which can only be done in the Driver (<a href="https://stackoverflow.com/questions/44495803/how-to-read-write-a-hive-table-from-within-the-spark-executors">discussed here</a>) so not in parallel. I would like to do something like this:</p>
  
  <pre><code># List all parquet files to be converted
  import glob, os
  files = glob.glob('/path/*.parquet'))
  
  sc = SparkSession.builder.getOrCreate()
  sc.parallelize(files, 2).foreach(lambda parquetFile: convert_parquet_to_tfrecord(parquetFile))
  </code></pre>
  
  <p>Could I construct the function <code>convert_parquet_to_tfrecord</code> that would be able to do this on the executors?</p>
  
  <p>I've also tried just using the wildcard when reading all the parquet files:</p>
  
  <pre><code>SQLContext(sc).read.parquet('/path/*.parquet')
  </code></pre>
  
  <p>This indeed reads all parquet files, but unfortunately not into individual partitions. It appears that the original structure gets lost, so it doesn't help me if I want the exact contents of the individual parquet files converted into TFrecord files. </p>
  
  <p>Any other suggestions?</p>