title: Load large flat file based data in warehouse from s3
body: <p>This is more of design and architecture question so please bare with me.</p>
  <p><strong>Requirement</strong> - Lets consider we have 2 type of flat files (csv, xls or txt) for below two db tables.</p>
  <pre><code>Doctor
     name
     degree
     ...
  Patient
     name
     doctorId
     age
     ...
  </code></pre>
  <p>each file contains data of each tables respectively. (volume 2-3millions each file).
  we have to load these two files data into <code>Doctor, Patient</code> table of warehouse. after some of the validations like <strong>null value, foreign key, duplicates in doctor</strong> etc..
  if any invalid data identifies, i will need to attach the reasons like null value, duplicate value. so that i can evaluate the invalid data.
  Note that, expectations is to load 1 million records in ~1-2mins of span.</p>
  <p><strong>My Designed workflow (so far)</strong></p>
  <p>After several articles and blog reading i find it to go with AWS Glue &amp; Databrew of my ETL for source to target along with custom validations.</p>
  <p>Please find below design and architecture. Suggest &amp; guide me on it.
  <strong>Is there any scope of parallel or partition based processing for quick validation and loading the data?</strong> Your help is going to really help me and others (who gonna come to this type of case).</p>
  <p>Thanks a ton.</p>
  <p><a href="https://i.stack.imgur.com/sAP1S.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/sAP1S.png" alt="enter image description here" /></a></p>