Much more Efficient way to Parse and Process Large files with Json Objects
Question: This is by far the craziest question I have asked on SO but I am going to give
it a shot in the hope of getting some advice about whether or not I am
leveraging the right tools and methods for processing large amounts of data
efficiently. I'm not necessarily looking for help on optimizing my code unless
there is something I am completely overlooking but essentially would just like
to know if I should be going with a different framework all together instead
of Python. I'm new enough to Python to not be completely sure if it is
possible to process large amount of data and store into DB much more
efficiently. Essentially the below implementation reads text files in a
directory with each text file containing 50K lines of json objects that need
to be parsed and read and then converted into a csv before loading into a
database. I hate using list containers and I am hoping that there is something
else I can research implementing in Python for doing this in a much much
better way. My initial thoughts are that I should be researching using
generators but not totally sure. The crazy concat part at the end is important
because it converts a comma separated list into it's own rows. [Converting
Column with string separated values into
rows](http://stackoverflow.com/questions/24419551/converting-column-with-
string-separated-values-into-rows)

    
    
      triggerZipFiles = glob.glob('*.zip')
      for triggerFiles in triggerZipFiles:
          with zipfile.ZipFile(triggerFiles, 'r') as myzip:
              for logfile in myzip.namelist():
                  datacc = []
                  zipcc = []
                  csvout = '{}_US.csv'.format(logfile[:-4])
                  f = myzip.open(logfile)
                  contents = f.readlines()
                  for line in contents:
                      try:
                          parsed = json.loads(line[:-2])
                          if "CC" in parsed['data']['weatherType'] and "US" in parsed['zipcodes']:
                              datacc.append(parsed['data'])
                              zipcc.append(parsed['zipcodes'])
                      except:
                          pass
                  if len(datacc) > 0:
                      df = pd.concat([pd.DataFrame(zipcc), pd.DataFrame(datacc)], axis=1)
                      df = pd.concat((pd.Series((v, row['key'], row['key'], row['key'],
                        row['key'], row['key'], row['key'], row['key'], row['key'],
                        row['key'], row['key'], row['key'], row['key'], row['key'],
                        row['key'], row['key'], row['key'], row['key'], row['key'],
                        row['key'], row['key'], row['key'], row['key'], row['key'],
                        row['key'], row['key'], row['key'], row['key'],
                        row['key'], row['key'], row['key'], row['key'],
                        row['key'], row['key'], row['key'], row['key'],
                        row['key'], row['key'], row['key'], row['key'], row['key'],
                        row['key'], row['key'], row['key'], row['key'],
                        row['key'], row['key'], row['key'], row['key'], row['key'],
                        row['key'], row['key'], row['key'], row['key'], row['key'], row['key'],
                        row['key'], row['key'], row['key'], row['key'], row['key'], row['key'],
                        row['key'], row['key'], row['key'], row['key'], row['key'], row['key'],
                        row['key'], row['key'], row['key'], row['key']), df.columns) for _,
                          row in df.iterrows() for v in row['US']), axis=1).T                
                      df.to_csv(csvout, header=None, index=False)
                  else:
                      pass
                  print datetime.now().strftime('%Y/%m/%d %H:%M:%S') + ": Finished: {}".format(logfile)
    


Answer: Firstly, _lines_ isn't a particularly useful metric when it comes to json!  
Secondly, you've got the right idea: you definitely want to do this chunk-
based (read/clean/dump each part separately).

I recommend using pandas' `read_json` function, it's much more efficient at
creating DataFrames (it doesn't create a temporary python dict), _see
the[reading in json section of the docs](http://pandas.pydata.org/pandas-
docs/dev/io.html#reading-json)._

  * If there not _actually_ json, then doing string manipulation to make them json is usually your best bet.*
  * If you have "oddly-shaped" json, then you can either [`json_normalize`](http://pandas.pydata.org/pandas-docs/dev/io.html#normalization) when reading, or parse the columns which contain multiple columns after reading in the DataFrame (e.g. using a Series string method or apply).

_*it's unclear what the actual format is, but often it doesn't take much to
make them into**valid** json._

Python bonus tip: if you're more that a few indentation levels deep consider
breaking it apart into more functions. (The obvious choice here is to have
`f1(logfile)` and `f2(line)`, but using descriptive names...)

