title: Are parquet file created with pyarrow vs pyspark compatible?
body: <p>I have to convert analytics data in JSON to parquet in two steps. For the large amounts of existing data I am writing a PySpark job and doing</p>
  
  <pre><code>df.repartition(*partitionby).write.partitionBy(partitionby).
      mode("append").parquet(output,compression=codec)
  </code></pre>
  
  <p>however for incremental data I plan to use AWS Lambda. Probably, PySpark would be an overkill for it, and hence I plan to use PyArrow for it (I am aware that it unnecessarily involves Pandas, but I couldn't find a better alternative). So, basically:</p>
  
  <pre><code>import pyarrow.parquet as pq
  pq.write_table(table, outputPath, compression='snappy',
      use_deprecated_int96_timestamps=True)
  </code></pre>
  
  <p>I wanted to know if the Parquet files written by both PySpark and PyArrow will be compatible (with respect to Athena)?</p>