title: Parsing json lines using pandas
body: <p>I have a large json file with millions of lines. This file also has some error messages in it. Below is the sample:</p>
  
  <pre><code>{"MEASUREMENT_1":"12345678","MEASUREMENT_2":"123456789012","MEASUREMENT_3":"MEASUREMENT_TYPE","MEASUREMENT_4":1111111111111,"MEASUREMENT_5":-1122,"MEASUREMENT_6":-2233,"MEASUREMENT_7":"123456789"}
  {"MEASUREMENT_1":"87654321","MEASUREMENT_2":"987654321098","MEASUREMENT_3":"MEASUREMENT_TYPE_2","MEASUREMENT_4":222222222222,"MEASUREMENT_5":-4455,"MEASUREMENT_6":-6677,"MEASUREMENT_7":"123456789"}
  [2015-12-02 02:00:02,530] WARN Reconnect due to socket error: null 
  [2015-12-02 02:00:02,633] WARN Reconnect due to socket error: null 
  </code></pre>
  
  <p>As expected, the below code throws a <code>ValueError</code> because of the error message line in the file.</p>
  
  <pre><code>#!/usr/bin/python3.5
  import pandas as pd # Version 0.21.0
  df = pd.read_json(file, lines=True)
  </code></pre>
  
  <p>As this is very large file, I have used <code>chunksize</code> and an expection, as below:</p>
  
  <pre><code>max_records = 1e5
  df = pd.read_json(file, lines=True, chunksize=max_records)
  filtered_data = pd.DataFrame() # Initialize the dataframe
  try:
     for df_chunk in df:
         filtered_data = pd.concat([filtered_data, df_chunk])
  except ValueError:
         print ('\nSome messages in the file cannot be parsed')
  </code></pre>
  
  <p>But the drawback of the above approach is it misses some of the lines. Is there any better way to do this? I went through the documentation of <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_json.html" rel="nofollow noreferrer">http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_json.html</a> but couldn't find anything that can ignore the unparsed lines. Can someone help? </p>