title: PySpark Processing Stream data and saving processed data to file
body: <p>I am trying to replicate a device that is streaming it's location's coordinates, then process the data and save it to a text file.
  I am using Kafka and Spark streaming (on pyspark),this is my architecture: </p>
  
  <p>1-Kafka producer emits data to a topic named test in the following string format :</p>
  
  <pre><code>"LG float LT float" example : LG 8100.25191107 LT 8406.43141483
  </code></pre>
  
  <p>Producer code : </p>
  
  <pre><code>from kafka import KafkaProducer
  import random
  
  producer = KafkaProducer(bootstrap_servers='localhost:9092')
  
  for i in range(0,10000):
      lg_value = str(random.uniform(5000, 10000))
      lt_value = str(random.uniform(5000, 10000))
  producer.send('test', 'LG '+lg_value+' LT '+lt_value)
  
  producer.flush()
  </code></pre>
  
  <p>The producer works fine and i get the streamed data in the consumer(and even in spark)</p>
  
  <p>2- Spark streaming is receiving this stream,i can even <code>pprint()</code> it</p>
  
  <p>Spark streaming processing code</p>
  
  <pre><code>from pyspark import SparkContext
  from pyspark.streaming import StreamingContext
  from pyspark.streaming.kafka import KafkaUtils
  
  ssc = StreamingContext(sc, 1)
  kvs = KafkaUtils.createDirectStream(ssc, ["test"], {"bootstrap.servers": "localhost:9092"})
  
  lines = kvs.map(lambda x: x[1])
  
  words      = lines.flatMap(lambda line: line.split(" "))
  words.pprint()
  word_pairs = words.map(lambda word: (word, 1))
  counts     = word_pairs.reduceByKey(lambda a, b: a+b)
  results    = counts.foreachRDD(lambda word: word.saveAsTextFile("C:\path\spark_test.txt"))
  //I tried this kvs.saveAsTextFiles('C:\path\spark_test.txt')
  // to copy all stream and it works fine
  ssc.start()
  ssc.awaitTermination()
  </code></pre>
  
  <p>As an error i get :</p>
  
  <pre><code>16/12/26 00:51:53 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
  org.apache.spark.SparkException: Python worker did not connect back in time
  </code></pre>
  
  <p>And other exceptions.</p>
  
  <p>What i actually want is to save each entry <code>"LG float LT float"</code> as a JSON format in a file,but first i want to simply save the coordinates in a file,i cant seem to make that happen.Any ideas?</p>
  
  <p>I can provide with the full stack trace if needed</p>