title: Spark/Scala - Validate JSON document in a row of a streaming DataFrame
body: <p>I have a streaming application which is processing a streaming DataFrame with column "body" that contains a JSON string.</p>
  
  <p>So in the body is something like (these are four input rows):</p>
  
  <pre><code>{"id":1, "ts":1557994974, "details":[{"id":1,"attr2":3,"attr3":"something"}, {"id":2,"attr2":3,"attr3":"something"}]}
  {"id":2, "ts":1557994975, "details":[{"id":1,"attr2":"3","attr3":"something"}, {"id":2,"attr2":"3","attr3":"something"},{"id":3,"attr2":"3","attr3":"something"}]}
  {"id":3, "ts":1557994976, "details":[{"id":1,"attr2":3,"attr3":"something"}, {"id":2,"attr2":3}]}
  {"id":4, "ts":1557994977, "details":[]}
  </code></pre>
  
  <p>I would like to check that each row has the correct schema (data types and contains all attributes). I would like to filter out and log the invalid records somewhere (like a Parquet file). I am especially interested in the "details" array - each of the nested documents must have specified fields and correct data types.</p>
  
  <p>So in the example above only row id = 1 is valid.</p>
  
  <p>I was thinking about a case class such as:</p>
  
  <pre><code>case class Detail(
    id: Int,
    attr2: Int,
    attr3: String
  )
  
  case class Input(
    id: Int,
    ts: Long,
    details: Seq[Detail]
  )
  </code></pre>
  
  <p>and Try but not sure how to go about it.</p>
  
  <p>Could someone help, please?</p>
  
  <p>Thanks</p>