title: How to Read Filename from S3 using AWS Glue ETL Tools
body: <p>I have a some files in S3 that look like this (all in the same path):</p>
  <pre><code>group1_20210415.csv
  group2_20210415.csv
  group1_20210416.csv
  group2_20210416.csv
  </code></pre>
  <p>The schema for each file is rather simple:</p>
  <pre><code>group1_name, group1_id
  group2_name, group2_id
  </code></pre>
  <p>I want to be able to query these names and ids from S3 with Athena, and use AWS Glue to crawl that S3 location when new files are present.</p>
  <p>Specifically, I want to have a table in Athena with schema:</p>
  <pre><code>group1_name, group1_id, group2_name, group2_id, hit_date
  </code></pre>
  <p>My intuition says to use AWS Glue PySpark to combine the data in the S3 files into a single DataFrame, which is simple enough. However, the date for each file exists in the file name itself not in the data.</p>
  <p>Is there a way to extract the 'date' part of the filename and use that as a column in the AWS Glue PySpark DataFrame? If not, does anyone's intuition suggest an alternative method?</p>