title: Load a large json file 3.7GB into dataframe and convert to csv file using ijson
body: <p>I have a large json data file with 3.7gb. Iam going to load the json file to dataframe and delete unused columns than convert it to csv and load to sql.
  ram is 40gb
  My json file structure</p>
  <pre><code>{&quot;a&quot;:&quot;Ho Chi Minh City, Vietnam&quot;,&quot;gender&quot;:&quot;female&quot;,&quot;t&quot;:&quot;841675194476&quot;,&quot;id&quot;:&quot;100012998502085&quot;,&quot;n&quot;:&quot;Lee Mến&quot;}
  {&quot;t&quot;:&quot;84945474479&quot;,&quot;id&quot;:&quot;100012998505399&quot;,&quot;n&quot;:&quot;Hoàng Giagia&quot;}
  {&quot;t&quot;:&quot;841679770421&quot;,&quot;id&quot;:&quot;100012998505466&quot;,&quot;n&quot;:&quot;Thoại Mỹ&quot;}
  </code></pre>
  <p>I try to load data but it fails because of out of memory</p>
  <pre><code>data_phone=[]
  with open('data.json', 'r', encoding=&quot;UTF-8&quot;) as f:
      numbers = ijson.items(f, 't',multiple_values=True)
      for num in numbers :
          data_phone.append(num)
  </code></pre>
  <p>It shows errors</p>
  <pre><code>Out of memory
  </code></pre>
  <p>I try another way</p>
  <p>import json
  fb_data={}
  i=1</p>
  <pre><code>with open('output.csv', 'w') as csv_file:
      with open(&quot;Vietnam_Facebook_Scrape.json&quot;, encoding=&quot;UTF-8&quot;) as json_file:
          for line in json_file:
              data = json.loads(line)
              try:
                  csv_file.write('; '.join([str(i),&quot;/&quot;,data[&quot;t&quot;],data[&quot;fbid&quot;]]))
              except:
                  pass
  </code></pre>
  <p>Then I convert from csv to sql, it still show error &quot;MemoryError:&quot;</p>
  <pre><code>con = db.connect(&quot;fbproject.db&quot;)
  cur = con.cursor()
  with open('output.csv', 'r',encoding=&quot;UTF-8&quot;) as csv_file:
      for item in csv_file:
          cur.execute('insert into fbdata values (?)', (item,))
  con.commit()
  con.close()
  </code></pre>
  <p>Thanks for reading</p>