How to load large csv file from web in python
Using requests

You can read the file line by line iteratively. You don't need to store the whole dataset into memory. This will only maintain a single row in memory at a time. This is very fast and memory efficient.

import requests
from contextlib import closing
import csv

url = "http://samplecsvs.s3.amazonaws.com/SalesJan2009.csv"

with closing(requests.get(url, stream=True)) as r:
    f = (line.decode('utf-8') for line in r.iter_lines())
    reader = csv.reader(f, delimiter=',', quotechar='"')
    for row in reader:
        print(row)




Using pandas

You can also do this with pandas, however this will store an entire chunk of the dataset in memory. However, this library is very powerful and can make your processing way easier.

import pandas as pd

url = "http://samplecsvs.s3.amazonaws.com/SalesJan2009.csv"
df = pd.read_csv(url)

print(df.head())