title: Read file in a folder using regex expression in pyspark
body: <p>I am trying to read a file from a folder using the below code in my local and s3. For now I am testing in my local.</p>
  <pre><code>#job_date_str = &quot;2021-08-03T05:20:00&quot;
  path = f'{config[&quot;input&quot;][&quot;logs&quot;][job]}{job_date_str}(.)*'
  # also tried path = f'{config[&quot;input&quot;][&quot;logs&quot;][job]}{job_date_str}*'
  dataframe = spark.read.schema(schema).json(path, multiLine=multiline)
  </code></pre>
  <p>The folder has file inside directory(config[&quot;input&quot;][&quot;logs&quot;][job]) as <strong>2021-08-03T05:20:00.000-31bnrrpUT5qNLtEX8zWt.log.gz. (YYYY-MM-DDTh:m:s.000-something)</strong></p>
  <p>It is throwing me error as</p>
  <pre><code>21/08/03 12:12:49 ERROR logger: error: java.net.URISyntaxException: Relative path in absolute URI: 2021-08-03T05:20:00.000-31bnrrpUT5qNLtEX8zWt.log.gz in get_json
  </code></pre>
  <p>not sure what does it mean.
  <strong>here I have file names starting with job_date_str</strong></p>
  <p>But earlier I was trying to read files using.</p>
  <pre><code>#job_date_str = &quot;2021-08-03T05:20:00&quot;
  path = f'{config[&quot;input&quot;][&quot;logs&quot;][job]}{job_date_str}/*'
  dataframe = spark.read.schema(schema).json(path, multiLine=multiline)
  </code></pre>
  <p>and It worked in local and s3. <strong>here I was storing files inside job_date_str</strong> directory.</p>