title: What is the fastest way to transform a very large JSON file with Spark?
body: <p>I am having a rather large JSON file (<a href="http://jmcauley.ucsd.edu/data/amazon/" rel="nofollow">Amazon product data</a>) with a lot of single JSON objects. Those JSON objects contain text that I want to preprocess for a specific training task but it is the preprocessing that I need to speed up here. One JSON object looks like this:</p>
  
  <pre><code>{
    "reviewerID": "A2SUAM1J3GNN3B",
    "asin": "0000013714",
    "reviewerName": "J. McDonald",
    "helpful": [2, 3],
    "reviewText": "I bought this for my husband who plays the piano.  He is having a wonderful time playing these old hymns.  The music  is at times hard to read because we think the book was published for singing from more than playing from.  Great purchase though!",
    "overall": 5.0,
    "summary": "Heavenly Highway Hymns",
    "unixReviewTime": 1252800000,
    "reviewTime": "09 13, 2009"
  }
  </code></pre>
  
  <p>The task would be to extract <code>reviewText</code> from each JSON object and perform some preprocessing like lemmatizing etc.</p>
  
  <p>My problem is that I don't know how I could use Spark in order to speed this task up on a cluster.. I am actually not even sure if I can read that JSON file as a stream object-by-object and parallelize the main task.</p>
  
  <p>What would be the best way to get started with this?</p>