title: spark: Attempted to use Broadcast after it was destroyed
body: <p>The following code works</p>
  
  <pre><code> @throws(classOf[IKodaMLException])
   def soMergeTarget1( oldTargetIdx: Double, newTargetIdx: Double): RDDLabeledPoint =
   {
     try
     {
      logger.trace("\n\n--sparseOperationRenameTargetsInNumeriOrder--\n\n")
      val oldTargetIdxb=spark.sparkContext.broadcast(oldTargetIdx)
      val newTargetIdxb=spark.sparkContext.broadcast(newTargetIdx)
  
      val newdata:RDD[(LabeledPoint,Int,String)] = sparseData.map
      {
        r =&gt;
  
          val currentLabel: Double = r._1.label
          currentLabel match
          {
            case x if x == oldTargetIdxb.value =&gt;
            val newtrgt=newTargetIdxb.value
            (new LabeledPoint(newtrgt, r._1.features), r._2, r._3)
            case _ =&gt; r
          }
      }
    val newtargetmap=ilp.targetMap.filter(e=&gt; !(e._2 == oldTargetIdx))
    oldTargetIdxb.destroy
    newTargetIdxb.destroy
    new RDDLabeledPoint(newdata,copyColumnMap,newtargetmap,ilp.name)
  }
  </code></pre>
  
  <p>But, having destroyed the broadcast variables at the end of the method, the <code>newtrgt</code> variable in the RDD is also destroyed. 
  The trouble is that once the RDD is returned from this method it could be used by any analyst in any code. So, I seem to have lost all control of the broadcast variables.</p>
  
  <p><strong>Questions:</strong> </p>
  
  <p>If I don't destroy the variables, will spark destroy them when reference to the RDD disappears?</p>
  
  <p>(Perhaps a naive question but....) I tried a little hack <code>val newtrgt=oldTargetIdxb.value + 1 -1</code> thinking that might create a new reference that is distinct from the broadcast variable. It didn't work. I must admit that surprised me. Can someone explain why the hack didn't work (I'm not suggesting it was a good idea, but I am curious).</p>