title: PySpark dataframe to_json() function
body: <p>I have a dataframe like below,</p>
  
  <pre><code>&gt;&gt;&gt; df.show(10,False)
  +-----+----+---+------+
  |id   |name|age|salary|
  +-----+----+---+------+
  |10001|alex|30 |75000 |
  |10002|bob |31 |80000 |
  |10003|deb |31 |80000 |
  |10004|john|33 |85000 |
  |10005|sam |30 |75000 |
  +-----+----+---+------+
  </code></pre>
  
  <p>Converting the entire row of df into one new column "jsonCol",</p>
  
  <pre><code>&gt;&gt;&gt; newDf1 = df.withColumn("jsonCol", to_json(struct([df[x] for x in df.columns])))
  &gt;&gt;&gt; newDf1.show(10,False)
  +-----+----+---+------+--------------------------------------------------------+
  |id   |name|age|salary|jsonCol                                                 |
  +-----+----+---+------+--------------------------------------------------------+
  |10001|alex|30 |75000 |{"id":"10001","name":"alex","age":"30","salary":"75000"}|
  |10002|bob |31 |80000 |{"id":"10002","name":"bob","age":"31","salary":"80000"} |
  |10003|deb |31 |80000 |{"id":"10003","name":"deb","age":"31","salary":"80000"} |
  |10004|john|33 |85000 |{"id":"10004","name":"john","age":"33","salary":"85000"}|
  |10005|sam |30 |75000 |{"id":"10005","name":"sam","age":"30","salary":"75000"} |
  +-----+----+---+------+--------------------------------------------------------+
  </code></pre>
  
  <p>Instead of converting the entire row into a JSON string like in the above step I needed a solution to select only few columns based on the value of the field. I have provided a sample condition in the below command.</p>
  
  <p>But when I started using the when function, the resultant JSON string's column names(keys) are gone. Only getting column names by their position, instead of the actual column names(keys) </p>
  
  <pre><code>&gt;&gt;&gt; newDf2 = df.withColumn("jsonCol", to_json(struct([ when(col(x)!="  ",df[x]).otherwise(None) for x in df.columns])))
  &gt;&gt;&gt; newDf2.show(10,False)
  +-----+----+---+------+---------------------------------------------------------+
  |id   |name|age|salary|jsonCol                                                  |
  +-----+----+---+------+---------------------------------------------------------+
  |10001|alex|30 |75000 |{"col1":"10001","col2":"alex","col3":"30","col4":"75000"}|
  |10002|bob |31 |80000 |{"col1":"10002","col2":"bob","col3":"31","col4":"80000"} |
  |10003|deb |31 |80000 |{"col1":"10003","col2":"deb","col3":"31","col4":"80000"} |
  |10004|john|33 |85000 |{"col1":"10004","col2":"john","col3":"33","col4":"85000"}|
  |10005|sam |30 |75000 |{"col1":"10005","col2":"sam","col3":"30","col4":"75000"} |
  +-----+----+---+------+---------------------------------------------------------+
  </code></pre>
  
  <p>I needed to use the when function but to have the results as in newDf1 with actual column names(keys). Can someone help me out?</p>