Python - How to remove duplicates in huge CSV file (200+ Million rows)
Two possible approaches:
SQL Select Distinct
Load the CSV file into a database and us SQL Select Distinct and export the query result as a Csv file. Some database let you define a CSV file as a table and let you run SQL queries on the CSV file
Select distinct URL from UrlTable
Sort the File
You could 

Sort the file on the URL using your operating systems sort command (both Windows and Linux have Sort commands).
Read the file and check if the URL is different from the last URL. You only need check the URL when it changes
We made an Image recognition API that accepts image URL and responds an image description. This process takes about 5-20 seconds.
I have a huge CSV file with 200+ million rows of image URLs scraped from different sources. I found that the CSV file has duplicate image URLs from different sources. So I don't need to send all the URLs to image recognition API, I need to send only unique URLs to the API but I need to populate response from API to all the rows.
I read the CSV file in chunks of 100K rows and created a set of unique Image URLs and process them in API and again populate the results back to the CSV file. But It ended up in memory issues (16 GB RAM), I can't able to create a set of unique Image URLs with 200+ million rows.
Any suggestion?