title: Write delta file to S3 (MinIO) - PySpark 2.4.3
body: <p>I am currently trying to write a <code>delta-lake</code> <code>parquet</code> file to S3, which I replace with a MinIO locally.</p>
  
  <p>I can perfectly fine read/write standard <code>parquet</code> files to <code>S3</code>.</p>
  
  <p>However, when I use the <a href="https://docs.delta.io/latest/quick-start.html" rel="nofollow noreferrer">delta lake example</a></p>
  
  <p>Configure <a href="https://docs.delta.io/latest/delta-storage.html#configuration-for-s3" rel="nofollow noreferrer">delta to s3</a></p>
  
  <p>It seems I can't write <code>delta_log/</code> to my <code>MinIO</code>.</p>
  
  <p>So I tried to set: <code>fs.AbstractFileSystem.s3a.impl</code> and <code>fs.s3a.impl</code>.</p>
  
  <p>I am using <code>pyspark[sql]==2.4.3</code> which I use in my current <code>venv</code>.</p>
  
  <p><code>src/.env</code>: </p>
  
  <pre class="lang-sh prettyprint-override"><code># pyspark packages
  DELTA = io.delta:delta-core_2.11:0.3.0
  HADOOP_COMMON = org.apache.hadoop:hadoop-common:2.7.3
  HADOOP_AWS = org.apache.hadoop:hadoop-aws:2.7.3
  PYSPARK_SUBMIT_ARGS = ${HADOOP_AWS},${HADOOP_COMMON},${DELTA}
  </code></pre>
  
  <p><code>src/spark_session.py</code>:</p>
  
  <pre class="lang-py prettyprint-override"><code># configure s3 connection for read/write operation (native spark)
  hadoop_conf = sc.sparkContext._jsc.hadoopConfiguration()
  hadoop_conf.set("fs.s3a.endpoint", self.aws_endpoint_url)
  hadoop_conf.set("fs.s3a.access.key", self.aws_access_key_id)
  hadoop_conf.set("fs.s3a.secret.key", self.aws_secret_access_key)
  # hadoop_conf.set("fs.AbstractFileSystem.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")  #  when using hadoop 2.8.5
  # hadoop_conf.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")  #  alternative to above hadoop 2.8.5
  hadoop_conf.set("fs.s3a.path.style.access", "true")
  hadoop_conf.set("spark.history.fs.logDirectory", 's3a://spark-logs-test/')
  </code></pre>
  
  <p><code>src/apps/raw_to_parquet.py</code></p>
  
  <pre class="lang-py prettyprint-override"><code># Trying to write pyspark dataframe to MinIO (S3)
  
  raw_df.coalesce(1).write.format("delta").save(s3_url)
  
  </code></pre>
  
  <hr>
  
  <p><code>bash</code>:</p>
  
  <pre class="lang-sh prettyprint-override"><code># RUN CODE
  spark-submit --packages $(PYSPARK_SUBMIT_ARGS) src/run_onlineretailer.py
  
  </code></pre>
  
  <p>Error with <code>hadoop-common: 2.7.3</code>, <code>hadoop-aws: 2.7.3</code>: <code>java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.fs.s3a.S3AFileSystem.&lt;init&gt;(java.net.URI, org.apache.hadoop.conf.Configuration)</code></p>
  
  <p>So with this error I then updated to <code>hadoop-common: 2.8.5</code>, <code>hadoop-aws: 2.8.5</code>, to fix the <code>NoSuchMethodException</code>. Because <code>delta</code> needs: <code>S3AFileSystem</code></p>
  
  <p><code>py4j.protocol.Py4JJavaError: An error occurred while calling o89.save.
  : java.lang.NoSuchMethodError: org.apache.hadoop.security.ProviderUtils.excludeIncompatibleCredentialProviders(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/Class;)Lorg/apache/hadoop/conf/Configuration</code></p>
  
  <p>So to me it seems that the <code>parquet</code> file can be written without a problem, however, delta creates these <code>delta_log</code> folder which cannot be recognised (I think?).</p>
  
  <p>Current <a href="https://github.com/Thelin90/berserkr/pull/4" rel="nofollow noreferrer">source code</a>.</p>
  
  <p>Read several different similar questions but nobody seem to have been trying with <code>delta lake</code> files.</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>It currently work with these settings:</p>
  
  <pre class="lang-sh prettyprint-override"><code>#pyspark packages
  DELTA_LOGSTORE = spark.delta.logStore.class=org.apache.spark.sql.delta.storage.S3SingleDriverLogStore
  DELTA = io.delta:delta-core_2.11:0.3.0
  HADOOP_COMMON = org.apache.hadoop:hadoop-common:2.7.7
  HADOOP_AWS = org.apache.hadoop:hadoop-aws:2.7.7
  PYSPARK_SUBMIT_ARGS = ${HADOOP_AWS},${HADOOP_COMMON},${DELTA}
  PYSPARK_CONF_ARGS = ${DELTA_LOGSTORE}
  </code></pre>
  
  <pre class="lang-py prettyprint-override"><code># configure s3 connection for read/write operation (native spark)
  hadoop_conf = sc.sparkContext._jsc.hadoopConfiguration()
  hadoop_conf.set("fs.s3a.endpoint", self.aws_endpoint_url)
  hadoop_conf.set("fs.s3a.access.key", self.aws_access_key_id)
  hadoop_conf.set("fs.s3a.secret.key", self.aws_secret_access_key)
  </code></pre>
  
  <pre class="lang-sh prettyprint-override"><code>spark-submit --packages $(PYSPARK_SUBMIT_ARGS) --conf $(PYSPARK_CONF_ARGS) src/run_onlineretailer.py
  </code></pre>
  
  <p>The weird thing is that it will only work like this.</p>
  
  <p>If I try to set it with <code>sc.conf</code> or <code>hadoop_conf</code> it does not work, see uncommented code:</p>
  
  <pre class="lang-py prettyprint-override"><code>def spark_init(self) -&gt; SparkSession:
  
      sc: SparkSession = SparkSession \
          .builder \
          .appName(self.app_name) \
          .config("spark.sql.warehouse.dir", self.warehouse_location) \
          .getOrCreate()
  
      # set log level
      sc.sparkContext.setLogLevel("WARN")
  
      # Enable Arrow-based columnar data transfers
      sc.conf.set("spark.sql.execution.arrow.enabled", "true")
  
      # sc.conf.set("spark.delta.logStore.class", "org.apache.spark.sql.delta.storage.S3SingleDriverLogStore") # does not work
  
      # configure s3 connection for read/write operation (native spark)
      hadoop_conf = sc.sparkContext._jsc.hadoopConfiguration()
      hadoop_conf.set("fs.s3a.endpoint", self.aws_endpoint_url)
      hadoop_conf.set("fs.s3a.access.key", self.aws_access_key_id)
      hadoop_conf.set("fs.s3a.secret.key", self.aws_secret_access_key)
      #hadoop_conf.set("spark.delta.logStore.class", "org.apache.spark.sql.delta.storage.S3SingleDriverLogStore") # does not work
  
      return sc
  </code></pre>
  
  <p>If somebody can explain this, it would be great. Is it because <code>.getOrCreate()</code>? Seems impossible to set the <code>conf</code> without this call? Except in the command line when running application.</p>