title: How to JSON-escape a String field in Spark dataFrame with new column
body: <p>How to write a new column with JSON format through DataFrame. I tried several approaches but it's writing the data as JSON-escaped String field.
  Currently its writing as
  <code>{"test":{"id":1,"name":"name","problem_field": "{\"x\":100,\"y\":200}"}}</code></p>
  
  <p>Instead I want it to be as 
  <code>{"test":{"id":1,"name":"name","problem_field": {"x":100,"y":200}}}</code></p>
  
  <p><code>problem_field</code> is a new column that is being created based on the values read from other fields as:</p>
  
  <pre><code>val dataFrame = oldDF.withColumn("problem_field", s)
  </code></pre>
  
  <p>I have tried the following approaches</p>
  
  <ol>
  <li><code>dataFrame.write.json(&lt;&lt;outputPath&gt;&gt;)</code></li>
  <li><code>dataFrame.toJSON.map(value =&gt; value.replace("\\", "").replace("{\"value\":\"", "").replace("}\"}", "}")).write.json(&lt;&lt;outputPath&gt;&gt;)</code></li>
  </ol>
  
  <p>Tried converting to <code>DataSet</code> as well but no luck. Any pointers are greatly appreciated.</p>
  
  <p>I have already tried the logic mentioned here: <a href="https://stackoverflow.com/questions/37225414/how-to-let-spark-parse-a-json-escaped-string-field-as-a-json-object-to-infer-the">How to let Spark parse a JSON-escaped String field as a JSON Object to infer the proper structure in DataFrames?</a></p>