title: spark.conf.set with SparkR
body: <p>I have a Databricks cluster running on Azure and want read / write data from Azure Data Lake Storage using <code>SparkR</code> / <code>sparklyr</code>. Therefore I configured the two <a href="https://docs.microsoft.com/en-us/azure/azure-databricks/databricks-extract-load-sql-data-warehouse" rel="nofollow noreferrer">resources</a>.</p>
  
  <p>Now I have to provide the Spark environment the necessary configurations to authenticate against the Data Lake Storage.</p>
  
  <p>Setting the configs using the <code>PySpark API</code> works:</p>
  
  <pre><code>    spark.conf.set("dfs.adls.oauth2.access.token.provider.type", "ClientCredential")
      spark.conf.set("dfs.adls.oauth2.client.id", "****")
      spark.conf.set("dfs.adls.oauth2.credential", "****")
      spark.conf.set("dfs.adls.oauth2.refresh.url", "https://login.microsoftonline.com/****/oauth2/token")
  </code></pre>
  
  <p>In the end <code>SparkR</code> / <code>sparklyr</code> should be used. Here I couldn't figure out where to set the <code>spark.conf.set</code>. I would have guessed something like:</p>
  
  <pre><code>    sparkR.session(
      sparkConfig = list(spark.driver.memory = "2g",
      spark.conf.set("dfs.adls.oauth2.access.token.provider.type", "ClientCredential"),
      spark.conf.set("dfs.adls.oauth2.client.id", "****"),
      spark.conf.set("dfs.adls.oauth2.credential", "****"),
      spark.conf.set("dfs.adls.oauth2.refresh.url", "https://login.microsoftonline.com/****/oauth2/token")
      ))
  </code></pre>
  
  <p>Would be awesome if one of the experts using the <code>SparkR</code> API could help me out here. Thanks!</p>
  
  <p>EDIT:
  The answer by user10791349 is correct and it works. Another solution is mounting the external <a href="https://docs.azuredatabricks.net/spark/latest/data-sources/azure/azure-storage.html" rel="nofollow noreferrer">data source</a> which is best practice. This is currently only possible using Scala or Python but the mounted data source is afterwards available using the SparkR API.</p>