title: Real-time pipeline for processing data, insert into PSQL
body: <p>I have files that will be coming in daily that I would like to process as they come in and insert into existing sql tables (using postgres). What is the best way to create an automated pipeline?</p>
  <p>I have already written the file processing scripts on python which return the data in format to be appended to the sql tables. What is the best way to make this pipeline real-time. That is, have the pipeline automatically process the file as its sent to me and then have the data added to the sql table. At the moment i am doing this manually by batch but i want to fully automate the process. The key missing step is having the file automatically processed by the scrip. I am reading Apache kafka can help but I'm still a novice here.</p>
  <p>Any help is much appreciated!</p>