title: PySpark - running processes
body: <p>In my Spark Streaming (Spark 2.1.0) application, I have the need to build a graph from a file and initializing that graph takes between 5 and 10 seconds.</p>
  
  <p>So I tried initializing it once per executor so it'll be initialized only once.</p>
  
  <p>After running the application, I've noticed that it's initiated much more than once per executor, every time with a different process id (every process has it's own logger).</p>
  
  <p>Doesn't every executor have it's own JVM and it's own process? Or is that only relevant when I develop in JVM languages like Scala/Java?
  Do executors in PySpark spawn new processes for new tasks?</p>
  
  <p>And if they do, how can I make sure that my graph object will really only be initiated once?</p>
  
  <p>The way I initialize my object:</p>
  
  <pre class="lang-py prettyprint-override"><code>class MySingletons(object):
      kafka_producer = KafkaProducer(bootstrap_servers=['localhost:9092'])
      tagger = _init_tagger()  # This returns an object with a graph inside of it
  </code></pre>
  
  <pre class="lang-py prettyprint-override"><code>    @classmethod
      def handle_batch(cls, records_batch):
          analyzed = cls.tagger.tag(records_batch)
          return analyzed
  </code></pre>
  
  <p>And then I call it inside the driver:</p>
  
  <pre class="lang-py prettyprint-override"><code>def handle_partition(records: Sequence):
      records_lst = list(records)
      if len(records_lst) &gt; 0:
          MySingletons.handle_batch(records_lst)
  
  
  def handle_rdd(rdd: RDD):
      rdd_values = rdd.map(lambda x: x[1])
      rdd_values.foreachPartition(handle_partition)
  
  
  ssc.union(*streams).filter(lambda x: x[1] is not None).foreachRDD(handle_rdd)
  ssc.start()
  ssc.awaitTermination()
  </code></pre>
  
  <p>Thanks :)</p>