title: Automate loading data from S3 to Redshift
body: <blockquote>
  <p>I want too load data from S3 to Redshift. The data coming to S3 in around 5MB{approximate size} per sec.</p>
  </blockquote>
  <p>I need to automate the loading of data from S3 to Redshift.
  The data to S3 is dumping from the kafka-stream consumer application.</p>
  <p>The folder S3 data is in folder structure.
  Example folder :
  <strong>bucketName/abc-event/2020/9/15/10</strong></p>
  <p>files in this folder :</p>
  <p>abc-event-2020-9-15-10-00-01-abxwdhf.  5MB</p>
  <p>abc-event-2020-9-15-10-00-02-aasdljc.  5MB</p>
  <p>abc-event-2020-9-15-10-00-03-thntsfv.  5MB</p>
  <blockquote>
  <p>the files in S3 have json objects separated with next line.</p>
  </blockquote>
  <blockquote>
  <p>This data need to be loaded to <strong>abc-event</strong> table in redshift.</p>
  </blockquote>
  <p>I know few options like <strong>AWS Data pipeline</strong>, <strong>AWS Glue</strong>, <strong>AWS Lambda Redshift loader</strong> (<a href="https://aws.amazon.com/blogs/big-data/a-zero-administration-amazon-redshift-database-loader/" rel="nofollow noreferrer">https://aws.amazon.com/blogs/big-data/a-zero-administration-amazon-redshift-database-loader/</a>).</p>
  <p>What would be the best way to do it.
  Really appreciate if someone will guide me.
  Thanks you</p>
  <p>=============================================</p>
  <blockquote>
  <p>Thanks Prabhakar for the answer. Need some help in continuation on this.</p>
  </blockquote>
  <p>Created a table in Data Catalog by crawler and
  then running a ETLL job in glue does the job of loading the data from S3 to redshift.</p>
  <p>I am using approach 1. Predicate pushdown
  New files get loaded in S3 in different partition say (new hour started.)</p>
  <p>I am adding new partition using a AWS Glue python script job.
  Adding new partition in the table using Athena API. (using ALTER TABLE ADD PARTITION).</p>
  <p>I have checked in the console that the new partition gets added by the python script job. I checked new partion gets added in Data catalog table.</p>
  <p>When I run the same job with pushdown predicate giving same partition added by the python script glue job.</p>
  <p>The job did not load the new files from S3 in this new partition to Redshift.</p>
  <p>I cant figure out what I am doing wrong ???</p>