title: Error Handling on mongoimport
body: <p>I have a directory of roughly 45,000 json files. The total size is around 12.8 GB currently. This is website data from kissmetrics and its structure is <a href="http://support.kissmetrics.com/apis/data/#what-is-the-format-of-the-data" rel="nofollow">detailed here</a>. </p>
  
  <p>The data:
  Each file is multiple json documents separated by a newline
  It will be updated every 12 hours with new additional files</p>
  
  <p>I want to import this data to mongoDB using mongoimport. I've tried this shell script to make the process easier: </p>
  
  <pre><code>for filename in revisions/*;
  
  do
  
  echo $filename
  mongoimport --host &lt;HOSTNAME&gt;:&lt;PORT&gt; --db &lt;DBNAME&gt; --collection &lt;COLLECTIONNAME&gt; \
      --ssl --sslCAFile ~/mongodb.pem --username &lt;USERNAME&gt; --password &lt;PASSWORD&gt; \
      --authenticationDatabase admin $filename
  
  done
  </code></pre>
  
  <p>This will have errors </p>
  
  <pre><code>2016-06-18T00:31:10.781+0000    using 1 decoding workers
  2016-06-18T00:31:10.781+0000    using 1 insert workers
  2016-06-18T00:31:10.781+0000    filesize: 113 bytes
  2016-06-18T00:31:10.781+0000    using fields:
  2016-06-18T00:31:10.822+0000    connected to: &lt;HOSTNAME&gt;:&lt;PORT&gt;
  2016-06-18T00:31:10.822+0000    ns: &lt;DBNAME&gt;.&lt;COLLECTION&gt;
  2016-06-18T00:31:10.822+0000    connected to node type: standalone
  2016-06-18T00:31:10.822+0000    standalone server: setting write concern w to 1
  2016-06-18T00:31:10.822+0000    using write concern: w='1', j=false, fsync=false, wtimeout=0
  2016-06-18T00:31:10.822+0000    standalone server: setting write concern w to 1
  2016-06-18T00:31:10.822+0000    using write concern: w='1', j=false, fsync=false, wtimeout=0
  2016-06-18T00:31:10.824+0000    Failed: error processing document #1: invalid character 'l' looking for beginning of value
  2016-06-18T00:31:10.824+0000    imported 0 documents
  </code></pre>
  
  <p>I will potentially run into this error, and from my inspection is not due to malformed data. </p>
  
  <p>The error may happen hours into the import. </p>
  
  <p><strong>Can I parse the error in mongoimport to retry the same document?</strong> I don't know if the error will have this same form, so I'm not sure if I can try to handle it in bash. <strong>Can I keep track of progress in bash and restart if terminated early?</strong> Any suggestions on importing large data of this size or handling the error in shell?</p>