title: Reading in very, very large NDJSON
body: <p>I have a 33GB NDJSON file I need to read into a data.table in R. It's gzipped into a 2GB file, ideally I would like to keep it compressed.</p>
  
  <p>The structure isn't so important except that (when imported via <code>jsonlite::stream_in</code>), the data I need are in only a few simple columns. The vast majority of the weight of the data is held in <code>list</code>s within three columns I want to discard as soon as possible.</p>
  
  <p>My two challenges are: how can I parallelize the read-in, and how can I limit memory usage (right now my worker on this file is using 175GB memory)?</p>
  
  <p>What I'm doing now:</p>
  
  <p><code>dt.x &lt;- data.table(flatten(stream_in(gzfile("source.gz"))[, -c(5:7)]))</code></p>
  
  <p>Ideas:</p>
  
  <p>Maybe there is some way to ignore a portion of the NDJSON during <code>stream_in</code>?</p>
  
  <p>Could I parse the <code>gzfile</code> connection, eg with regex, before it goes to <code>stream_in</code>, to remove the excess data?</p>
  
  <p>Can I do something like <code>readLines</code> on the <code>gzfile</code> connection to read the data 1 million lines per worker?</p>
  
  <p>EDIT: If at all possible, my goal is to make this portable to other users and keep it entirely within R.</p>