title: Read Amazon S3 specific folder from Azure databricks without public access to the bucket
body: <p>Problem : Access specific directory contents present in an Amazon S3 bucket from databricks(pyspark)</p>
  <p>Bucket policies setup</p>
  <ol>
  <li>S3 bucket is not publicly accessible</li>
  <li>Directory inside S3 bucket is granted permission for public access and contains parquet files.</li>
  </ol>
  <p>Databricks setup on Azure</p>
  <ol>
  <li>Databricks run time version 8.3</li>
  <li>Spark version 3.1.1</li>
  </ol>
  <p>Code snippet :</p>
  <ol>
  <li>Set access key and secret key as</li>
  </ol>
  <pre class="lang-py prettyprint-override"><code>sc._jsc.hadoopConfiguration().set(&quot;fs.s3n.awsAccessKeyId&quot;, access_key)
  sc._jsc.hadoopConfiguration().set(&quot;fs.s3n.awsSecretAccessKey&quot;, secret_key)
  </code></pre>
  <ol start="2">
  <li>Read data in a dataframe from the directory</li>
  </ol>
  <pre class="lang-py prettyprint-override"><code>AWS_S3_BUCKET_NAME = BUCKET_NAME
   
  read_df = spark.read \
      .format(&quot;parquet&quot;) \
      .option(&quot;header&quot;, &quot;true&quot;) \
      .load(&quot;s3n://{}/{}&quot;.format(AWS_S3_BUCKET_NAME, directory_name))
  </code></pre>
  <p>access_key, secret_key, BUCKET_NAME and directory_name are proper values.</p>
  <blockquote>
  <p>Error message : Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden;</p>
  </blockquote>
  <p>Are there any properties to setup so as to read the specific directory contents from S3 bucket using Azure databricks when the S3 bucket is not publicly accessible?</p>