title: Pyspark streaming count predictions of each class
body: <p>I am making a classifier in spark streaming but I am having two issues with broadcasted variables and accumulators.</p>
  <ol>
  <li>When loading the trained pipeline, I get a pickle error. I am using the code below. When loading the model in the foreachRDD function it works fine. Any ideas what I am doing wrong here?</li>
  </ol>
  <pre><code>def getModel(sparkContext):
      if (&quot;model&quot; not in globals()):
          # read model
          model = PipelineModel.load(&quot;path&quot;)
          
          globals()[&quot;model&quot;] = sparkContext.broadcast(model)
      return globals()[&quot;model&quot;]
  </code></pre>
  <ol start="2">
  <li>I have an accumulator for the total number of predictions made which works but I am also trying to count the number of times each class is predicted. I was trying to use this function but I am getting an error that I am trying to broadcast an RDD.</li>
  </ol>
  <pre><code>def getAccumulatorClass(sparkContext, label):
      name = &quot;accum_class_&quot; + label
      if (name not in globals()):
          globals()[name] = sparkContext.accumulator(0)
      return globals()[name]
  
  def process_stream(rdd):
      # code for predicting classes
      classified_by_class = df_classified.groupBy('prediction').count()
      count_classes = udf(lambda pred, count: getAccumulatorClass(rdd.context, pred).add(int(count))) 
      classified_by_class.select(count_classes(col(&quot;prediction&quot;), col(&quot;count&quot;)))
  
  fileStream.foreachRDD(process_stream)
  </code></pre>
  <p>The content of the classified by class is:</p>
  <p><a href="https://i.stack.imgur.com/CVPRN.png" rel="nofollow noreferrer">content of classified by class</a></p>
  <p>Thank you for any help/pointers in solving these issues!</p>