title: Unable to write data from Spark to Azure Blob Storage - generic Microsoft error message
body: <p>Getting a generic Microsoft error <strong>'<code>Caused by: com.microsoft.azure.storage.StorageException: One of the request inputs is not valid.</code>'</strong> when using park-shell to write data to Azure Storage blob. </p>
  
  <p>I am using Spark 2.4.4 (2.7 Hadoop Distribution) distribution and copied the following 2 Azure specific files to <strong>$SPARK_INSTALL_HOME/jars</strong> folder:</p>
  
  <ul>
  <li>hadoop-azure-2.7.3.jar</li>
  <li>azure-storage-7.0.0.jar</li>
  </ul>
  
  <p>Connect to spark shell after copying azure specific files:
  <em>> bin/spark-shell --jars jars/hadoop-azure-2.7.3.jar,jars/azure-storage-7.0.0.jar</em></p>
  
  <p>Create Spark Context along with appropriate Hadoop configuration parameters:</p>
  
  <pre><code>scala&gt; sc
  scala&gt; sc.hadoopConfiguration.set("fs.azure", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
  scala&gt; sc.hadoopConfiguration.set("fs.azure.account.key.ACCOUNT_NAME.blob.core.windows.net", "ACCOUNT_KEY")*
  </code></pre>
  
  <p>Read data from Azure Blob storage successfully:</p>
  
  <pre><code>scala&gt; val df = spark.read.format("csv").option("inferSchema", "true").load("wasbs://CONTAINER_NAME@ACCOUNT_NAME.blob.core.windows.net/public_datasets/orders")*
  
  scala&gt; df.show()
  +-----+----+---+--------+------+---------+                                      
  |  _c0| _c1|_c2|     _c3|   _c4|      _c5|
  +-----+----+---+--------+------+---------+
  |31614|1499| EF|54662084|3666.0|Household|
  |31615|1500| GH| 4727745|4119.0|Household|
  |31616|1501| OP|40921163|2292.0|Household|
  |31617|1502| IJ|11066508|7338.0|Household|
  |31618|1503| OP|11958569|3171.0|Household|
  |31619|1504| UV|29212069| 619.0|Household|
  |31620|1504| WX|69836336|4887.0|         |
  |31621|1506| UV|28518402|9498.0|     Loan|
  |31622|1507| ST|  156510|6619.0|Household|
  |31623|1508| OP|10600459|1657.7|  Leasing|
  |31624|1508| UV| 9055811|8967.0|Household|
  |31625|1509| EF|43322950|4943.0|Household|
  |31626|1509| QR|20525896|2320.0|         |
  |31627|1510| OP|82537191|9985.0|Household|
  |31628|1511| OP|43543106|1657.7|  Leasing|
  |31629|1512| YZ|42362171|8822.0|Household|
  |31630|1513| AB|52238965|1632.0|Household|
  |31631|1513| QR|85800695|3912.0|         |
  |31632|1513| OP|59476637|1316.0|Insurance|
  |31633|1513| GH|65290647| 689.0|         |
  +-----+----+---+--------+------+---------+
  only showing top 20 rows
  </code></pre>
  
  <p>Write data back to a different location:</p>
  
  <pre><code>df.write.mode("overwrite").csv("wasbs://CONTAINER_NAME@ACCOUNT_NAME.blob.core.windows.net/public_datasets/orders_modified/")
  </code></pre>
  
  <p>The following exception is being thrown:</p>
  
  <pre><code>   Error committing the output of task: attempt_20200608142341_0006_m_000000_8
     org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: One of the request inputs is not valid.
         at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2482)
         at org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.execute(NativeAzureFileSystem.java:424)
         at org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1997)
         at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:531)
         at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:502)
         at org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:50)
         at org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:77)
         at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitTask(HadoopMapReduceCommitProtocol.scala:225)
         at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:78)
         at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247)
         at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
         at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
         at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248)
         at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
         at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
         at org.apache.spark.scheduler.Task.run(Task.scala:123)
         at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
         at java.lang.Thread.run(Thread.java:748)
     Caused by: com.microsoft.azure.storage.StorageException: One of the request inputs is not valid.
         at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
         at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:305)
         at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:175)
         at com.microsoft.azure.storage.blob.CloudBlob.startCopy(CloudBlob.java:883)
         at com.microsoft.azure.storage.blob.CloudBlob.startCopyFromBlob(CloudBlob.java:825)
         at org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobWrapperImpl.startCopyFromBlob(StorageInterfaceImpl.java:399)
         at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2449)
         ... 22 more
     20/06/08 14:23:45 ERROR Utils: Aborting task
     org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: One of the request inputs is not valid.
         at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2482)
         at org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.execute(NativeAzureFileSystem.java:424)
         at org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1997)
         at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:531)
         at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:502)
         at org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:50)
         at org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:77)
         at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitTask(HadoopMapReduceCommitProtocol.scala:225)
         at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:78)
         at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247)
         at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
         at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
         at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248)
         at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
         at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
         at org.apache.spark.scheduler.Task.run(Task.scala:123)
         at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
         at java.lang.Thread.run(Thread.java:748)
     Caused by: com.microsoft.azure.storage.StorageException: One of the request inputs is not valid.
         at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
         at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:305)
         at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:175)
         at com.microsoft.azure.storage.blob.CloudBlob.startCopy(CloudBlob.java:883)
         at com.microsoft.azure.storage.blob.CloudBlob.startCopyFromBlob(CloudBlob.java:825)
         at org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobWrapperImpl.startCopyFromBlob(StorageInterfaceImpl.java:399)
         at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2449)
         ... 22 more
  
  </code></pre>
  
  <p>Any suggestions to get past this problem will be highly appreciated?</p>