title: Parallel Processing AWS S3 Data in Python
body: <p>I have a list of files that I need to access and process from S3 buckets through a lambda function and the idea is to loop through each of the files and collect data from all files in parallel. My first thought was to use threading which resulted in an issue that only allowed my max pool size to be 10, whereas I'm processing many files. I want to be able to continuously append processes until all files have been accessed instead of creating a list of processes and then running them in parallel which seems to be the case in multiprocessing's Pool. I'd appreciate any suggestions.</p>