title: Parse Complex JSON in SCALA using spray-json
body: <p>I am trying to parse a JSON string to a case class in Scala (So I can do the filtering data processing etc).
  After some research I am going with <a href="https://github.com/spray/spray-json" rel="nofollow">spray-json</a> as there are several examples on the link. Unfortunately the link does not show how to parse a JSON with nested fields that have arrays.</p>
  
  <p>I am testing my code on a Scala notebook with the code below and it works.</p>
  
  <pre><code>// Dependencies
  io.spray spray-json_2.10 1.3.2
  import spray.json._
  import DefaultJsonProtocol._ // if you don't supply your own Protocol (see below)
  
  // simple source
  val source = """{
    "EventId": "29ca61f3-b8b6-41e7-8236-802fa232e7cf",
    "Timestamp": "2016-03-09T20:14:07.5535193Z",
    "StartTime": "2016-03-09T02:51:04.397",
    "EndTime": "2016-03-09T02:51:04.397",
    "ActiveStates": "{\"No Motion\":1,\"Motion Detected\":1,\"Face Detected\":1}",
    "Created": "2016-03-09T02:51:04.397",
    "Modified": "2016-03-09T02:51:04.397"
  }"""
  
  // simple case class
  case class claX(  EventId: String,
    Timestamp: String,
    StartTime: String,
    EndTime: String,
    ActiveStates: String,
    Created: String,
    Modified: String)
  
  
  object MyJsonProtocol extends DefaultJsonProtocol {
    implicit val claXFormat = jsonFormat7(claX)
  }
  import MyJsonProtocol._
  import spray.json._
  
  
  val json = source.parseJson // parse string to json
  val cx0 = json.convertTo[claX] // convert to class claX
  </code></pre>
  
  <p>My problem is when the JSON string has nested array in data that has a nested 'Product' class inside it. This is the sample JSON:</p>
  
  <pre><code>{
        "EventId": "29ca61f3-b8b6-41e7-8236-802fa232e7cf",
        "Timestamp": "2016-03-09T20:14:07.5535193Z",
        "StartTime": "2016-03-09T02:51:04.397",
        "EndTime": "2016-03-09T02:51:04.397",
        "ActiveStates": "{\"No Motion\":1,\"Motion Detected\":1,\"Face Detected\":1}",
        "Created": "2016-03-09T02:51:04.397",
        "Modified": "2016-03-09T02:51:04.397",
        "Data": {
          "AgeRange": {
            "Name": "30 - 35"
          },
          "Company": {
            "Id": "f3ad1744-0ead-458a-9416-852c43ccde24"
          },
          "CompanyType": {
            "Name": "Retailer"
          },
          "ConnectorType": {
            "Name": "Camera Capturing"
          },
          "Content": {
            "Ids": [
              "0c0f0a9a-fece-4b3e-abb4-0f508d357220"
            ]
          },
          "Customer": {
            "LoyaltyId": 0
          },
          "DeviceRegistries": [
            {
              "Id": "f19f5daa-e9b9-43d0-91a7-51da4fdd0e31",
              "DeviceName": "Company 3 Cooler",
              "DeviceType": "CCU"
            }
          ],
          "Emotion": {
            "Name": "Happy"
          },
          "Gender": {
            "Name": "Male"
          },
          "Products": [
            {
              "Name": "Molson Canadian",
              "ProductCategory": "Beverage",
              "InventoryTrackingNumberType": "SKU",
              "InventoryTrackingNumber": "438654935776",
              "ProductPrice": {
                "RetailPrice": 2.1,
                "RetailPriceSymbol": "?",
                "PromotionPrice": 1.8,
                "PromotionPriceSymbol": "?"
              }
            },
            {
              "Name": "Coors Original",
              "ProductCategory": "Beverage",
              "InventoryTrackingNumberType": "SKU",
              "InventoryTrackingNumber": "438654935775",
              "ProductPrice": {
                "RetailPrice": 1.1,
                "RetailPriceSymbol": "?",
                "PromotionPrice": 0.8,
                "PromotionPriceSymbol": "?"
              }
            },
            {
              "Name": "Coors Light",
              "ProductCategory": "Beverage",
              "InventoryTrackingNumberType": "SKU",
              "InventoryTrackingNumber": "438654935778",
              "ProductPrice": {
                "RetailPrice": 6.1,
                "RetailPriceSymbol": "?",
                "PromotionPrice": 5.8,
                "PromotionPriceSymbol": "?"
              }
            },
            {
              "Name": "Blue Moon",
              "ProductCategory": "Beverage",
              "InventoryTrackingNumberType": "SKU",
              "InventoryTrackingNumber": "438654935777",
              "ProductPrice": {
                "RetailPrice": 4.1,
                "RetailPriceSymbol": "?",
                "PromotionPrice": 3.8,
                "PromotionPriceSymbol": "?"
              }
            }
          ],
          "Race": {
            "Name": "Latin"
          },
          "Region": {
            "Name": "Region 01"
          },
          "SensorRegistry": {
            "Name": "Company 3 Camera 01"
          },
          "SensorType": {
            "Name": "Proximity"
          },
          "SfuRegistries": {
            "Ids": [
              "7effea8c-56dd-4905-bbc3-2158d14cd7cc",
              "24a7253d-174a-44f0-8145-483cc0f45adb",
              "bc970c8e-7e41-4889-859b-55c6a3f8ba5d",
              "46e599f5-8082-499f-b5d0-9d611409a652"
            ]
          },
          "Shelves": {
            "Ids": [
              "ea442504-7d64-4c01-bdde-1eb46e53b81c",
              "d6fe9c78-e21b-4a57-b620-99a7d94d46f9"
            ]
          },
          "State": {
            "Name": "Face Detected"
          },
          "StockLevel": {
            "OnHand": 0
          },
          "Store": {
            "Id": "268c852d-86b8-4b7c-b865-2f29a3e2307e"
          },
          "Unit": {
            "Id": "52c58781-b2bf-46ea-81ad-b9d9fbacb471"
          },
          "UnitType": {
            "Name": "5-Shelf Cooler"
          }
        },
        "id": "54bfd971-0fec-4e0e-87cc-851a697705e9"
      }
  </code></pre>
  
  <p>I've made two more case classes to manage the 'Product' and the 'Prices'</p>
  
  <pre><code>case class ProductPrice(RetailPrice: Double,
            RetailPriceSymbol: Double,
            PromotionPrice: Double,
            PromotionPriceSymbol: Double)
  
  case class Product(Name: String,
          ProductCategory: String,
          InventoryTrackingNumberType: String,
          InventoryTrackingNumber: String,
          ProductPrice: ProductPrice)
  </code></pre>
  
  <p>What I don't know is how do you combine this so that the Data node in the JSON is parsed properly into a claXBig (where everything from the JSON string is correctly parsed.
  This is where I am tripping up:</p>
  
  <pre><code>case class claX2(  EventId: String,
    Timestamp: String,
    StartTime: String,
    EndTime: String,
    ActiveStates: String,
    Created: String,
    Modified: String,
    Data: Map[String, Any]) // &lt;- how do I parse this and the nested products
  
  object MyJsonProtocol2 extends DefaultJsonProtocol {
    implicit val claXFormat2 = jsonFormat8(claX2)
  }
  </code></pre>
  
  <p>I'm also trying to load a larger JSON (collection of these 'events') using code outlined <a href="https://github.com/sryza/aas/blob/master/ch08-geotime/src/main/scala/com/cloudera/datascience/geotime/GeoJson.scala" rel="nofollow">here</a></p>
  
  <p>So I added a new case class below to handle the array of 'event's or claX2</p>
  
  <pre><code>case class claX2Collection(clax2s: Array[claX2])
      extends IndexedSeq[claX2] {
    def apply(index: Int) = clax2s(index) //&lt;- not sure what this mean 
    def length = clax2s.length // or whether index is doing anything
  }
  </code></pre>
  
  <p>I assume claX2Collection is correct as its compiling. But the code below is definitely wrong, but is needed to load the event collection from the JSON array</p>
  
  <pre><code>implicit object claX2JsonFormat extends RootJsonFormat[claX2]{
  def write(f: claX2) = {
  val buf = scala.collection.mutable.ArrayBuffer(
  "events" -&gt; JsString("claX2"), // &lt;- error
  "Timestamp" -&gt; JsObject(f.Timestamp), // error
  "StartTime" -&gt; JsObject(f.StartTime), // error
  "EndTime" -&gt; JsObject(f.EndTime), // error
  "ActiveStates" -&gt; JsObject(f.ActiveStates), // error
  "Created" -&gt; JsObject(f.Created), // errors
  "Modified" -&gt; JsObject(f.Modified), // errors
  "Data" -&gt; JsObject(f.Data) // errors
  )
  }
  def read(value:JsValue) = {
  val jso = value.asJsObject
  // not sure what to do here but
  // assuming I have to pick out
  val EventId = jso.fields.get("EventId")
  Timestamp = jso.fields.get("Timestamp")
  StartTime = jso.fields.get("StartTime")
  EndTime = jso.fields.get("EndTime")
  ActiveStates = jso.fields.get("ActiveStates")
  Created = jso.fields.get("Created")
  Modified = jso.fields.get("Modified")
  Data = jso.fields.get("Data")
  claX2(EventId,Timestamp,StartTime,EndTime,ActiveStates,Created,
  Modified,Data)
  }
  }
  </code></pre>
  
  <p>When that is fixed it should be able to read this JSON:</p>
  
  <pre><code>    {
    "type": "EventCollection",
  
  "events": [
  {
    "EventId": "29ca61f3-b8b6-41e7-8236-802fa232e7cf",
    "Timestamp": "2016-03-09T20:14:07.5535193Z",
    "StartTime": "2016-03-09T02:51:04.397",
    "EndTime": "2016-03-09T02:51:04.397",
    "ActiveStates": "{\"No Motion\":1,\"Motion Detected\":1,\"Face Detected\":1}",
    "Created": "2016-03-09T02:51:04.397",
    "Modified": "2016-03-09T02:51:04.397",
    "Data": {
      "AgeRange": {
        "Name": "30 - 35"
      },
      "Company": {
        "Id": "f3ad1744-0ead-458a-9416-852c43ccde24"
      },
      "CompanyType": {
        "Name": "Retailer"
      },
      "ConnectorType": {
        "Name": "Camera Capturing"
      },
      "Content": {
        "Ids": [
          "0c0f0a9a-fece-4b3e-abb4-0f508d357220"
        ]
      },
      "Customer": {
        "LoyaltyId": 0
      },
      "DeviceRegistries": [
        {
          "Id": "f19f5daa-e9b9-43d0-91a7-51da4fdd0e31",
          "DeviceName": "Company 3 Cooler",
          "DeviceType": "CCU"
        }
      ],
      "Emotion": {
        "Name": "Happy"
      },
      "Gender": {
        "Name": "Male"
      },
      "Products": [
        {
          "Name": "Molson Canadian",
          "ProductCategory": "Beverage",
          "InventoryTrackingNumberType": "SKU",
          "InventoryTrackingNumber": "438654935776",
          "ProductPrice": {
            "RetailPrice": 2.1,
            "RetailPriceSymbol": "?",
            "PromotionPrice": 1.8,
            "PromotionPriceSymbol": "?"
          }
        },
        {
          "Name": "Coors Original",
          "ProductCategory": "Beverage",
          "InventoryTrackingNumberType": "SKU",
          "InventoryTrackingNumber": "438654935775",
          "ProductPrice": {
            "RetailPrice": 1.1,
            "RetailPriceSymbol": "?",
            "PromotionPrice": 0.8,
            "PromotionPriceSymbol": "?"
          }
        },
        {
          "Name": "Coors Light",
          "ProductCategory": "Beverage",
          "InventoryTrackingNumberType": "SKU",
          "InventoryTrackingNumber": "438654935778",
          "ProductPrice": {
            "RetailPrice": 6.1,
            "RetailPriceSymbol": "?",
            "PromotionPrice": 5.8,
            "PromotionPriceSymbol": "?"
          }
        },
        {
          "Name": "Blue Moon",
          "ProductCategory": "Beverage",
          "InventoryTrackingNumberType": "SKU",
          "InventoryTrackingNumber": "438654935777",
          "ProductPrice": {
            "RetailPrice": 4.1,
            "RetailPriceSymbol": "?",
            "PromotionPrice": 3.8,
            "PromotionPriceSymbol": "?"
          }
        }
      ],
      "Race": {
        "Name": "Latin"
      },
      "Region": {
        "Name": "Region 01"
      },
      "SensorRegistry": {
        "Name": "Company 3 Camera 01"
      },
      "SensorType": {
        "Name": "Proximity"
      },
      "SfuRegistries": {
        "Ids": [
          "7effea8c-56dd-4905-bbc3-2158d14cd7cc",
          "24a7253d-174a-44f0-8145-483cc0f45adb",
          "bc970c8e-7e41-4889-859b-55c6a3f8ba5d",
          "46e599f5-8082-499f-b5d0-9d611409a652"
        ]
      },
      "Shelves": {
        "Ids": [
          "ea442504-7d64-4c01-bdde-1eb46e53b81c",
          "d6fe9c78-e21b-4a57-b620-99a7d94d46f9"
        ]
      },
      "State": {
        "Name": "Face Detected"
      },
      "StockLevel": {
        "OnHand": 0
      },
      "Store": {
        "Id": "268c852d-86b8-4b7c-b865-2f29a3e2307e"
      },
      "Unit": {
        "Id": "52c58781-b2bf-46ea-81ad-b9d9fbacb471"
      },
      "UnitType": {
        "Name": "5-Shelf Cooler"
      }
    },
    "id": "54bfd971-0fec-4e0e-87cc-851a697705e9"
  },
  {
    "EventId": "29ca61f3-b8b6-41e7-8236-802fa232e7cf",
    "Timestamp": "2016-03-09T20:14:07.5535193Z",
    "StartTime": "2016-03-09T02:51:04.397",
    "EndTime": "2016-03-09T02:51:04.397",
    "ActiveStates": "{\"No Motion\":1,\"Motion Detected\":1,\"Face Detected\":1}",
    "Created": "2016-03-09T02:51:04.397",
    "Modified": "2016-03-09T02:51:04.397",
    "Data": {
      "AgeRange": {
        "Name": "30 - 35"
      },
      "Company": {
        "Id": "f3ad1744-0ead-458a-9416-852c43ccde24"
      },
      "CompanyType": {
        "Name": "Retailer"
      },
      "ConnectorType": {
        "Name": "Camera Capturing"
      },
      "Content": {
        "Ids": [
          "0c0f0a9a-fece-4b3e-abb4-0f508d357220"
        ]
      },
      "Customer": {
        "LoyaltyId": 0
      },
      "DeviceRegistries": [
        {
          "Id": "f19f5daa-e9b9-43d0-91a7-51da4fdd0e31",
          "DeviceName": "Company 3 Cooler",
          "DeviceType": "CCU"
        }
      ],
      "Emotion": {
        "Name": "Happy"
      },
      "Gender": {
        "Name": "Male"
      },
      "Products": [
        {
          "Name": "Molson Canadian",
          "ProductCategory": "Beverage",
          "InventoryTrackingNumberType": "SKU",
          "InventoryTrackingNumber": "438654935776",
          "ProductPrice": {
            "RetailPrice": 2.1,
            "RetailPriceSymbol": "?",
            "PromotionPrice": 1.8,
            "PromotionPriceSymbol": "?"
          }
        },
        {
          "Name": "Coors Original",
          "ProductCategory": "Beverage",
          "InventoryTrackingNumberType": "SKU",
          "InventoryTrackingNumber": "438654935775",
          "ProductPrice": {
            "RetailPrice": 1.1,
            "RetailPriceSymbol": "?",
            "PromotionPrice": 0.8,
            "PromotionPriceSymbol": "?"
          }
        },
        {
          "Name": "Coors Light",
          "ProductCategory": "Beverage",
          "InventoryTrackingNumberType": "SKU",
          "InventoryTrackingNumber": "438654935778",
          "ProductPrice": {
            "RetailPrice": 6.1,
            "RetailPriceSymbol": "?",
            "PromotionPrice": 5.8,
            "PromotionPriceSymbol": "?"
          }
        },
        {
          "Name": "Blue Moon",
          "ProductCategory": "Beverage",
          "InventoryTrackingNumberType": "SKU",
          "InventoryTrackingNumber": "438654935777",
          "ProductPrice": {
            "RetailPrice": 4.1,
            "RetailPriceSymbol": "?",
            "PromotionPrice": 3.8,
            "PromotionPriceSymbol": "?"
          }
        }
      ],
      "Race": {
        "Name": "Latin"
      },
      "Region": {
        "Name": "Region 01"
      },
      "SensorRegistry": {
        "Name": "Company 3 Camera 01"
      },
      "SensorType": {
        "Name": "Proximity"
      },
      "SfuRegistries": {
        "Ids": [
          "7effea8c-56dd-4905-bbc3-2158d14cd7cc",
          "24a7253d-174a-44f0-8145-483cc0f45adb",
          "bc970c8e-7e41-4889-859b-55c6a3f8ba5d",
          "46e599f5-8082-499f-b5d0-9d611409a652"
        ]
      },
      "Shelves": {
        "Ids": [
          "ea442504-7d64-4c01-bdde-1eb46e53b81c",
          "d6fe9c78-e21b-4a57-b620-99a7d94d46f9"
        ]
      },
      "State": {
        "Name": "Face Detected"
      },
      "StockLevel": {
        "OnHand": 0
      },
      "Store": {
        "Id": "268c852d-86b8-4b7c-b865-2f29a3e2307e"
      },
      "Unit": {
        "Id": "52c58781-b2bf-46ea-81ad-b9d9fbacb471"
      },
      "UnitType": {
        "Name": "5-Shelf Cooler"
      }
    },
    "id": "54bfd971-0fec-4e0e-87cc-851a697705e9"
  }
  ]
  }
  </code></pre>