title: Q: Converting Avro to Parquet in Memory
body: <p>I am receiving Avro records from Kafka. I want to convert these records into Parquet files. I am following this blog post: <a href="http://blog.cloudera.com/blog/2014/05/how-to-convert-existing-data-into-parquet/" rel="nofollow">http://blog.cloudera.com/blog/2014/05/how-to-convert-existing-data-into-parquet/</a></p>
  
  <p>The code so far looks roughly like this: </p>
  
  <pre><code>final String fileName
  SinkRecord record, 
  final AvroData avroData
  
  final Schema avroSchema = avroData.fromConnectSchema(record.valueSchema());
  CompressionCodecName compressionCodecName = CompressionCodecName.SNAPPY;
  
  int blockSize = 256 * 1024 * 1024;
  int pageSize = 64 * 1024;
  
  Path path = new Path(fileName);
  writer = new AvroParquetWriter&lt;&gt;(path, avroSchema, compressionCodecName, blockSize, pageSize);
  </code></pre>
  
  <p>Now, this will do the Avro to Parquet conversion, but it will write the Parquet file to the disk. I was wondering if there was an easier way to just keep the file in memory so that I don't have to manage temp files on the disk. Thank you</p>