title: Python - How to remove duplicates in huge CSV file (200+ Million rows)
body: <p>We made an Image recognition API that accepts image URL and responds an image description. This process takes about 5-20 seconds.
  I have a huge CSV file with 200+ million rows of image URLs scraped from different sources. I found that the CSV file has duplicate image URLs from different sources. So I don't need to send all the URLs to image recognition API, I need to send only unique URLs to the API but I need to populate response from API to all the rows.</p>
  
  <p>I read the CSV file in chunks of 100K rows and created a set of unique Image URLs and process them in API and again populate the results back to the CSV file. But It ended up in memory issues (16 GB RAM), I can't able to create a set of unique Image URLs with 200+ million rows.</p>
  
  <p>Any suggestion?</p>