title: Writing RDD Data to CSV with Dynamic Columns in Spark - Scala
body: <p>I am reading multiple files from an HDFS directory, and for each file the generated data is printed using:</p>
  
  <pre><code>frequencies.foreach(x =&gt; println(x._1 + ": "+x._2))
  </code></pre>
  
  <p>And the printed data is (for <em>File1.txt</em>):</p>
  
  <pre><code>'text': 45
  'data': 100
  'push': 150
  </code></pre>
  
  <p>The key can be different for other files like (<em>File2.txt</em>):</p>
  
  <pre><code>'data': 45
  'lea': 100
  'jmp': 150
  </code></pre>
  
  <p>The key is not necessarily the same in all the files. I want all the file data to be written to a .csv file in the following format:</p>
  
  <pre><code>Filename   text  data  push  lea  jmp
  File1.txt  45    100   150   0    0
  File2.txt  0     45    0     100  150  ....
  </code></pre>
  
  <p>Can someone please help me find a solution to this problem?</p>