title: How to read/write protocol buffer messages with Apache Spark?
body: <p>I want to Read/write protocol buffer messages from/to HDFS with Apache Spark. I found these suggested ways:</p>
  
  <p>1) Convert protobuf messsages to Json with Google's Gson Library and then read/write them by SparkSql. This solution is explained in <a href="https://forums.databricks.com/questions/129/how-do-i-use-sparksql-with-protocol-buffers.html" rel="noreferrer">this link</a> But I think doing that (convert to json) is an extra task.</p>
  
  <p>2) Convert to Parquet file. There are <a href="https://github.com/apache/parquet-mr" rel="noreferrer"><strong>parquet-mr</strong></a> and <a href="https://github.com/saurfang/sparksql-protobuf" rel="noreferrer"><strong>sparksql-protobuf</strong></a>  github projects for this way but I don't want parquet file because I always work with all columns (not some columns) and in this way Parquet Format does not give me any gain (at least I think).</p>
  
  <p>3) <a href="https://github.com/scalapb/ScalaPB" rel="noreferrer">ScalaPB</a>. May be it's what I am looking for. but in scala language that I don't know anything about it. I am looking for a java-based solution. <a href="https://www.youtube.com/watch?v=lSq_6Ws16gw" rel="noreferrer">This youtube video</a> introduce scalaPB and explain how to use it (for scala developers).</p>
  
  <p>4) Through the use of the sequence file and this is what I looking for, but found nothing about that. So, my question is: How can I write protobuf messages to sequence file on HDFS and from that? Any other suggestion will be useful. </p>
  
  <p>5) Through twitter's <a href="https://github.com/twitter/elephant-bird" rel="noreferrer">Elephant-bird</a> Library. </p>