title: Reading a large csv from a S3 bucket using python pandas in AWS Sagemaker
body: <p>I'm trying to load a large CSV (~5GB) into pandas from S3 bucket.</p>
  
  <p>Following is the code I tried for a small CSV of 1.4 kb :</p>
  
  <pre><code>client = boto3.client('s3') 
  obj = client.get_object(Bucket='grocery', Key='stores.csv')
  body = obj['Body']
  csv_string = body.read().decode('utf-8')
  df = pd.read_csv(StringIO(csv_string))
  </code></pre>
  
  <p>This works well for a small CSV, but my requirement of loading a 5GB csv to pandas dataframe cannot be achieved through this (probably due to memory constraints when loading the csv by StringIO). </p>
  
  <p>I also tried below code</p>
  
  <pre><code>s3 = boto3.client('s3')
  obj = s3.get_object(Bucket='bucket', Key='key')
  df = pd.read_csv(obj['Body'])
  </code></pre>
  
  <p>but this gives below error.</p>
  
  <pre><code>ValueError: Invalid file path or buffer object type: &lt;class 'botocore.response.StreamingBody'&gt;
  </code></pre>
  
  <p>Any help to resolve this error is much appreciated.</p>