title: Dataproc failed to read parquet file in google cloud storage
body: <p>I have a parquet file in google cloud storage, then try to read it as below:</p>
  
  <pre><code>val parquetFile = sqlContext.read.parquet("gs://eng_sandbox1/shaw/testparquet/part-r-00000-b4aecbee-724e-40ea-b868-95f7e3f758a7.gz.parquet")
  </code></pre>
  
  <p>However, I encountered the following exception. One thing I noticed is gs path is different in that exception: bucket name "eng_sandbox1" is missed. Thanks!  </p>
  
  <pre><code>16/03/02 02:32:18 ERROR org.apache.spark.scheduler.TaskSetManager: Task 0 in stage 30.0 failed 4 times; aborting job
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 30.0 failed 4 times, most recent failure: Lost task 0.3 in stage 30.0 (TID 270, shaw-cluster-1-w-0.c.fabled-triumph-700.internal): java.io.FileNotFoundException: File not found : gs:/shaw/testparquet/part-r-00000-b4aecbee-724e-40ea-b868-95f7e3f758a7.gz.parquet
          at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.getFileStatus(GoogleHadoopFileSystemBase.java:1173)
          at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:385)
          at org.apache.parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:157)
          at org.apache.parquet.hadoop.ParquetRecordReader.initialize(ParquetRecordReader.java:140)
          at org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.&lt;init&gt;(SqlNewHadoopRDD.scala:180)
          at org.apache.spark.rdd.SqlNewHadoopRDD.compute(SqlNewHadoopRDD.scala:126)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
          at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
          at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
          at org.apache.spark.scheduler.Task.run(Task.scala:89)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
  
  Driver stacktrace:
          at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
          at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
          at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
          at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
          at scala.Option.foreach(Option.scala:236)
          at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
          at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
          at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)
          at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:927)
          at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
          at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
          at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
          at org.apache.spark.rdd.RDD.collect(RDD.scala:926)
          at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:166)
          at org.apache.spark.sql.execution.SparkPlan.executeCollectPublic(SparkPlan.scala:174)
          at org.apache.spark.sql.DataFrame$$anonfun$org$apache$spark$sql$DataFrame$$execute$1$1.apply(DataFrame.scala:1538)
          at org.apache.spark.sql.DataFrame$$anonfun$org$apache$spark$sql$DataFrame$$execute$1$1.apply(DataFrame.scala:1538)
          at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)
          at org.apache.spark.sql.DataFrame.withNewExecutionId(DataFrame.scala:2125)
          at org.apache.spark.sql.DataFrame.org$apache$spark$sql$DataFrame$$execute$1(DataFrame.scala:1537)
          at org.apache.spark.sql.DataFrame$$anonfun$org$apache$spark$sql$DataFrame$$collect$1.apply(DataFrame.scala:1542)
          at org.apache.spark.sql.DataFrame$$anonfun$org$apache$spark$sql$DataFrame$$collect$1.apply(DataFrame.scala:1542)
          at org.apache.spark.sql.DataFrame.withCallback(DataFrame.scala:2138)
          at org.apache.spark.sql.DataFrame.org$apache$spark$sql$DataFrame$$collect(DataFrame.scala:1542)
          at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:1519)
          at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:38)
          at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:43)
          at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:45)
          at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:47)
          at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:49)
          at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:51)
          at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:53)
          at $iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:55)
          at $iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:57)
          at $iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:59)
          at $iwC$$iwC.&lt;init&gt;(&lt;console&gt;:61)
          at $iwC.&lt;init&gt;(&lt;console&gt;:63)
          at &lt;init&gt;(&lt;console&gt;:65)
          at .&lt;init&gt;(&lt;console&gt;:69)
          at .&lt;clinit&gt;(&lt;console&gt;)
          at .&lt;init&gt;(&lt;console&gt;:7)
          at .&lt;clinit&gt;(&lt;console&gt;)
          at $print(&lt;console&gt;)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
          at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)
          at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
          at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
          at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
          at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
          at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
          at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
          at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
          at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
          at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
          at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
          at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
          at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
          at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
          at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
          at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
          at org.apache.spark.repl.Main$.main(Main.scala:31)
          at org.apache.spark.repl.Main.main(Main.scala)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
          at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
          at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
          at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
          at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  Caused by: java.io.FileNotFoundException: File not found : gs:/shaw/testparquet/part-r-00000-b4aecbee-724e-40ea-b868-95f7e3f758a7.gz.parquet
          at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.getFileStatus(GoogleHadoopFileSystemBase.java:1173)
          at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:385)
          at org.apache.parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:157)
          at org.apache.parquet.hadoop.ParquetRecordReader.initialize(ParquetRecordReader.java:140)
          at org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.&lt;init&gt;(SqlNewHadoopRDD.scala:180)
          at org.apache.spark.rdd.SqlNewHadoopRDD.compute(SqlNewHadoopRDD.scala:126)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
          at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
          at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
          at org.apache.spark.scheduler.Task.run(Task.scala:89)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
  </code></pre>