title: Calling pyspark function asynchronously with concurrent.futures
body: <p>I am trying to call python functions which use pyspark rdd objects methods and are time-consuming which blocks my application.
  I need to write it in an async fashion so that my app doesn't get blocked.
  Here is a miniature version of the actual thing I want to do.</p>
  
  <pre><code>from concurrent.futures import Future
  from pyspark import SparkContext
  
  sc = SparkContext()
  
  def add(a, b):
      f = Future()
      c = a + b
      d = a*b
      t = (c,d)
      rdd = sc.parallelize([t])
      f.set_result(rdd)
      # return rdd
  
  if __name__ == '__main__':
  
      f1 = add(90,8)
      f2 = add(8, 89)
  
      while (not f1.done()) and (not f2.done()):
          pass
  
      print(f1.result())
      print(f2.result())
  </code></pre>
  
  <p>I know the above code won't work straight away. How can I modify it, so that it will work ?</p>