title: Parquet backed table corrupted - HIVE - expected magic number at tail [80, 65, 82, 49] but found [1, 92, 78, 10]
body: <p>Distribution: CDH-4.6.0-1.cdh4.6.0.p0.26
  Hive Version: 0.10.0
  Parquet Version: 1.2.5</p>
  
  <p>I have two big date partitioned external Hive tables full of log files that I recently converted to Parquet to take advantage of the compression and columnar storage.  So far I've been very happy with the performance.</p>
  
  <p>Our dev team recently added a field to the logs, so I was charged with adding a column to both log tables.  It worked perfectly for one, but the other appears to have become corrupted.  I reverted the change, but I still can't query the table.</p>
  
  <p>I'm convinced the data is fine (because it didn't change) but something is wrong in the metastore.  An msck repair table repopulates the partitions after I drop/create, but does not take care of the error below.  There are two things that can fix it, but neither are making me happy:</p>
  
  <ol>
  <li>Re-insert the data.</li>
  <li>Copy data back into the table from the production cluster.</li>
  </ol>
  
  <p>I'm really hoping there's a command that I don't know about that will fix the table without needing to resort to the above 2 options.  Like I said, the data is fine.  I've googled the heck out of the error, and I get some results but they all pertain to Impala which is NOT what were using.</p>
  
  <pre><code>select * from upload_metrics_hist where dt = '2014-07-01' limit 5;
  </code></pre>
  
  <p>The problem is this:</p>
  
  <p><b>Caused by: java.lang.RuntimeException: hdfs://hdfs-dev/data/prod/upload-metrics/upload_metrics_hist/dt=2014-07-01/000005_0 is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [1, 92, 78, 10]</b></p>
  
  <p>Full error</p>
  
  <pre><code>2014-07-17 02:00:48,835 WARN org.apache.hadoop.mapred.Child: Error running child
  java.io.IOException: java.lang.reflect.InvocationTargetException
  at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
  at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
  at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:372)
  at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.&lt;init&gt;(HadoopShimsSecure.java:319)
  at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:433)
  at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
  at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:396)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
  at org.apache.hadoop.mapred.Child.main(Child.java:262)
  Caused by: java.lang.reflect.InvocationTargetException
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
  at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:358)
  ... 10 more
  Caused by: java.lang.RuntimeException: hdfs://hdfs-dev/data/prod/upload-metrics/upload_metrics_hist/dt=2014-07-01/000005_0 is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [1, 92, 78, 10]
  at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:263)
  at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:229)
  at parquet.hive.DeprecatedParquetInputFormat$RecordReaderWrapper.getSplit(DeprecatedParquetInputFormat.java:327)
  at parquet.hive.DeprecatedParquetInputFormat$RecordReaderWrapper.&lt;init&gt;(DeprecatedParquetInputFormat.java:204)
  at parquet.hive.DeprecatedParquetInputFormat.getRecordReader(DeprecatedParquetInputFormat.java:108)
  at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.&lt;init&gt;(CombineHiveRecordReader.java:65)
  ... 15 more
  </code></pre>