title: Parquet build with HDFS getmerge recovery
body: <p>The only post that touch a bit the subject is <a href="https://stackoverflow.com/questions/31108123/collecting-parquet-data-from-hdfs-to-local-file-system">here</a> but it does not solve my problem.</p>
  
  <p>Here is the problem we collect parquet to a local backup with:</p>
  
  <pre><code>$ hadoop fs -getmerge /dir/on/hdfs /local/dir
  </code></pre>
  
  <p>The error made is that we thought that the parquet multiple files organization is due to HDFS writing but we did not understood it was really the parquet file "normal" organization. So (not very much cleverly) we used getmerge of HDFS to do backup. The problem is that our data have been erased and we are now struggling recovering it.</p>
  
  <p>When analyzing (and reading the <a href="https://parquet.apache.org/documentation/latest/" rel="nofollow noreferrer">doc</a>) the parquet we found that all the files are originally made of chunk that contains data+metadata in between magic number 'PAR1' and added to this are given 2 - _metadata and _common_metadata - files of metadata.</p>
  
  <p>By observing that the getmerge process files in order (of the original parquet directory on hdfs), I came up with a script that take the data between 2 'PAR1' and make of it a chunk file. 
  The first 2 files built are (_common_metadata, _metadata).</p>
  
  <pre><code>filePrefix='part-'
  finalFilePrefix='part-r-'
  
  awk 'NR%2==0{ print $0 &gt; "part-"i++ }' RS='PAR1' $1
  
  nbFiles=$(ls -lah | grep 'part-' | wc -l)
  
  for num in $(seq 0 $nbFiles)
          do 
          fileName="$filePrefix$num"
          lastName=""
          if [ "$num" -eq "0" ]; then
                  lastName="_common_metadata"
                  awk '{print "PAR1" $0 "PAR1"}' $fileName &gt; $lastName
          else    
  
                  if [ "$num" -eq "1" ]; then
                          lastName="_metadata"
                          awk '{print "PAR1" $0 "PAR1"}' $fileName &gt; $lastName
                  else    
                          if [ -e $fileName ]; then
                                  count=$( printf "%05d" $(($num-2)) )
                                  lastName="$finalFilePrefix$count.gz.parquet"
                                  awk '{print "PAR1" $0 "PAR1"}' $fileName &gt; $lastName
                          fi      
                  fi              
          fi
          echo $lastName
          truncate --size=-1 $lastName
          rm -f "$fileName"
  done
  
  mv $1 $1.backup
  mkdir $1
  mv _* $1
  mv part* $1
  </code></pre>
  
  <p>Some observation about the script:</p>
  
  <ol>
  <li>It takes a "getmerge" parquet file in parameter</li>
  <li>All the part created are moved to a directory named after the original file (the later being renamed filename.backup)</li>
  <li>A byte must be taken at the end of each file - truncate - this is empirically made as the spark sc.load.parquet() could not read metadata files otherwise)</li>
  <li>Eventually we use hadoop fs -put to upload it to hdfs.</li>
  <li>Try to load it as a Dataframe
  As I said the _metadata (and _common_metadate files apparently) are read ok but still we have and error when loading the chunks:</li>
  </ol>
  
  <p>The code:</p>
  
  <pre><code>val newDataDF = sqlContext.read.parquet("/tmp/userActionLog2-leclerc-culturel-2016.09.04.parquet")
  newDataDF.take(1)
  </code></pre>
  
  <p>The error:</p>
  
  <pre><code>newDataDF: org.apache.spark.sql.DataFrame = [bson: binary]
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0  (TID 5, hdp-node4.affinytix.com): java.io.IOException: can not read class    org.apache.parquet.format.PageHeader: don't know what type: 13
  at org.apache.parquet.format.Util.read(Util.java:216)
  at org.apache.parquet.format.Util.readPageHeader(Util.java:65)
  at  org.apache.parquet.hadoop.ParquetFileReader$WorkaroundChunk.readPageHeader(ParquetFileReader.java:668)
  at org.apache.parquet.hadoop.ParquetFileReader$Chunk.readAllPages(ParquetFileReader.java:546)
  at org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:496)
  at org.apache.spark.sql.execution.datasources.parquet.UnsafeRowParquetRecordReader.checkEndOfRowGroup(UnsafeRowParquetRecordReader.java:604)
  at org.apache.spark.sql.execution.datasources.parquet.UnsafeRowParquetRecordReader.loadBatch(UnsafeRowParquetRecordReader.java:218)
  at org.apache.spark.sql.execution.datasources.parquet.UnsafeRowParquetRecordReader.nextKeyValue(UnsafeRowParquetRecordReader.java:196)
  at org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.hasNext(SqlNewHadoopRDD.scala:194)
  at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
  at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
  at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:308)
  at scala.collection.Iterator$class.foreach(Iterator.scala:727)
  at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
  at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
  at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
  at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
  at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
  at scala.collection.AbstractIterator.to(Iterator.scala:1157)
  at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
  at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
  at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
  at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212)
  at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1881)
  at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1881)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
  at org.apache.spark.scheduler.Task.run(Task.scala:89)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
  Caused by: parquet.org.apache.thrift.protocol.TProtocolException: don't know what type: 13
  at parquet.org.apache.thrift.protocol.TCompactProtocol.getTType(TCompactProtocol.java:806)
  at parquet.org.apache.thrift.protocol.TCompactProtocol.readFieldBegin(TCompactProtocol.java:500)
  at org.apache.parquet.format.InterningProtocol.readFieldBegin(InterningProtocol.java:158)
  at org.apache.parquet.format.PageHeader.read(PageHeader.java:828)
  at org.apache.parquet.format.Util.read(Util.java:213)
  ... 32 more
  </code></pre>
  
  <p>Given that our data are at stake here, if somebody had any idea that could help, I warmly thank him(er) in advance.</p>
  
  <p>Bye </p>