title: How to pivot a pyspark streaming dataframe
body: <p>I receive streaming data in pyspark structured streaming and I need to pivot them such that I can have a single row from that data. </p>
  
  <p>The struct of data coming to my cluster is that: </p>
  
  <pre><code>{
  "version": 1.0.0,
  "message": {
     "data": [{
      "name": "name_1", 
      "value": 1.0},
      ...
     {
      "name": "name_2", 
      "value": 2.0}]
   }
  }
  </code></pre>
  
  <p>My code is the following: </p>
  
  <pre><code>dfStreaming = spark \
    .readStream \
    .format("eventhubs") \
    .options(**optionConf()) \
    .load() \
    .select(explode("message.data").alias("data")) \
    .select(("data.*")) \
  </code></pre>
  
  <p>I get the following result dataframe: </p>
  
  <pre><code>|---------------------|------------------|
  |         Name        |       Value      |
  |---------------------|------------------|
  |        Name_1       |         1.0      |
  |---------------------|------------------|
  |        Name_2       |         2.0      |
  |---------------------|------------------|
  </code></pre>
  
  <p>But I need the following structure (it's actually a pivot of the table): </p>
  
  <pre><code>|---------------------|------------------|
  |        Name_1       |      Name_2      |
  |---------------------|------------------|
  |         1.0         |        2.0       |
  |---------------------|------------------|
  </code></pre>
  
  <p>The pivot on streaming dataframe is not permitted, but there should be a solution for that I suppose. </p>
  
  <p>Thank you so much for your help. </p>