title: Convert PySpark data frame to JSON with each column as a key
body: <p>I'm working on <code>PySpark</code>. I have a data frame which I need to dump as a <code>JSON</code> file but the the <code>JSON</code> file should have the following format for example - </p>
  
  <pre><code>{"Column 1": [9202, 9202, 9202, ....], "Column 2": ["FEMALE", "No matching concept", "MALE", ....]}
  </code></pre>
  
  <p>So there should be 1 key for each column and the corresponding value should have a list of all the values in that column</p>
  
  <p>I tried to convert this to a Pandas data frame and then convert to a dict before dumping it as a JSON and was successful in doing that but as the data volume is very I want to do it directly on the PySpark data frame </p>