title: Python CSV to JSON: Why is JSON significantly larger than the CSV file and how can it be fixed?
body: <p>I've run into an interesting problem when converting a CSV file to JSON.  I generate a CSV file based on output from an SQLite query and write this to the hard drive.</p>
  
  <p>I then load the CSV file into my script using Pandas:</p>
  
  <pre><code>import pandas as pd
  
  df_var  = pd.read_csv('input.csv',header=0, low_memory=False)
  </code></pre>
  
  <p>I used the <em>low_memory</em> option since the dataframe consists of multiple datatypes.  I am not opposed to storing everything as a string value if anyone suggests that.</p>
  
  <p>Each row of the dataframe represents one record (each record has 144 fields/columns), so I strip off one row at a time and feed it into a class I built to convert that row into a dictionary (this is inside the class):</p>
  
  <pre><code>class CSVParser:
  
  
      def __init__(self, recordid, tjson={}):
  
          self.recordid = recordid
          self.json     = tjson
  
      def create_json(self, df):
  
          o_dict = {
                      'root': {
                          'id': {
                              'field1':df['field1'],
                              'field2':df['field2'],
                              'field3':self.recordid,
                          },
  
                          'core': {
                              'field1':df['field1'],
                              'field2':df['field2'],
                              'field3':df['field3'],
                              'field4':df['field4'],
                              'field5':df['field5'],
                              'field6':df['field6'],
                          },
  
                          #REMAINING RECORDS LEFT OUT FOR BREVITY
  
                      }
                  }
  
          self.json.append(o_dict)
  </code></pre>
  
  <p>From here I write the JSON to disk</p>
  
  <pre><code>def write_json(self):
      #if self is not empty write JSON to file
  
      if self.json:
          filename = 'output/' + self.recordid + '_output.json'
          with open(filename,'w') as outfile:
              json.dump(self.json,outfile, indent=4, separators=(',', ': '))
          print('JSON saved to drive')
      else:
          print('\nEmpty JSON\n')
          return(self.json)
  </code></pre>
  
  <p>Here's what I get then, suppose I run just 500 records the CSV file will be around 23MB in size with the resulting JSON at ~190MB!  Surely I am adding some unintentional formatting that I just can't seem to find.  My belief was the JSON file would yield a smaller file size than the CSV.</p>
  
  <p>Last bit of info, I originally used an OrderedDict to run this but when I saw the resulting JSON file size I thought maybe the OrderedDict was adding some formatting that was increasing the size a lot, after moving back to a dictionary not much changed with the end file size.</p>
  
  <p>Please let me know if you need any other information and I'll be sure to provide it.</p>
  
  <p>What are your thoughts?</p>