title: Using scala to dump result processed by Spark to HDFS
body: <p>I'm a bit confused to find the right way to save data into HDFS after processing them with spark.</p>
  
  <p>This is what I'm trying to do. I'm calculating min, max and SD of numeric fields. My input files have millions of rows, but output will have only around 15-20 fields. So, the output is a single value(scalar) for each field. </p>
  
  <p>For example: I will load all the rows of FIELD1 into an RDD, and at the end, I will get 3 single values for FIELD 1(MIN, MAX, SD). I concatenated these three values into temporary string. In the end, I will have 15 to twenty rows, containing 4 columns in this following format</p>
  
  <pre><code>FIELD_NAME_1  MIN  MAX  SD
  FIELD_NAME_2  MIN  MAX  SD
  </code></pre>
  
  <p>This is a snippet of the code:</p>
  
  <pre><code>//create rdd
  val data = sc.textFile("hdfs://x.x.x.x/"+args(1)).cache()
  //just get the first column
  val values = data.map(_.split(",",-1)(1))
  
  val data_double= values.map(x=&gt;if(x==""){0}else{x}.toDouble)
  val min_value= data_double.map((_,1)).reduceByKey((_+_)).sortByKey(true).take(1)(0)._1
  val max_value= data_double.map((_,1)).reduceByKey((_+_)).sortByKey(false).take(1)(0)._1
  val SD = data_double.stdev
  </code></pre>
  
  <p>So, i have 3 variables, min_value, max_value and SD that I want to store back to hdfs. </p>
  
  <p>Question 1:
  Since the output will be rather small, do I just save it locally on the server? or should I dump it to HDFS. Seems to me like dumping the file locally makes better sense. </p>
  
  <p>Question 2:
  In spark, I can just call the following to save an RDD into text file</p>
  
  <pre><code>some_RDD.saveAsTextFile("hdfs://namenode/path")
  </code></pre>
  
  <p>How do I accomplish the same thing in for a String variable that is not an RDD in scala? should I parallelize my result into an RDD first and then call saveAsTextFile?</p>