title: Is there a way to merge multiple CSV files uploaded to AWS S3 bucket using Python?
body: <p>I need to setup an AWS Lambda function that triggers when new CSV files are uploaded to an S3 bucket to merge the CSV files into one Master file (they will have the same number of columns and column names), then that new Master file is uploaded to another S3 bucket.</p>
  
  <p>I'm using Python for the Lambda function. I created a zip folder with my Lambda function and the dependencies I used (Pandas and Numpy) and uploaded that. </p>
  
  <p>Currently I have to include the CSV files that I want merged together in the zip folder itself, the function merges those CSV files and the output (Master file) is in the logs, when I check in CloudWatch.</p>
  
  <p>I don't know how to link my code to the S3 buckets for input and output.</p>
  
  <p>This is for an app I'm working on.</p>
  
  <p>here's the python code I'm using:</p>
  
  <pre><code>    import os
      import glob
      import numpy
      import pandas as pd
  
      def handler(event, context):
          #find all csv files in the folder
          #use glob pattern matching -&gt; extension = 'csv'
          #save result in list -&gt; all_filenames
          extension = 'csv'
          all_filenames = [i for i in glob.glob('*.{}'.format(extension))]
  
          #combine all files in the list
          combined_csv = pd.concat([pd.read_csv(f) for f in all_filenames ])
  
          #export to csv
          combined_csv.to_csv( "/tmp/combined_csv.csv", index=False, encoding='utf-8-sig')
          f = open("/tmp/combined_csv.csv", "r")
          print(f.read())
          f.close()
  </code></pre>
  
  <p>I would like to not have to manually input the CSV files in the same zip folder as my python script every time, and also have the output Master CSV file be in a separate S3 bucket.</p>