title: Closing PySpark notebook properly
body: <p>I'm using Jupyter notebook with PySpark, which uses Spark as a kernel.</p>
  
  <p>The problem is that I'm not sure how to properly close it and I have an impression that something keeps hanging, as the memory on the driver on which the notebook is running gets full and crashes (I get GC overhead exception).</p>
  
  <p>I'm closing the whole thing by simply killing the notebook using the process id which I save to .pid file. But I have a feeling that the following state is note good:</p>
  
  <p><a href="https://i.stack.imgur.com/9xW5B.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/9xW5B.png" alt="problem"></a></p>
  
  <p>What is the actual problem and how to solve it, that is, how the close the whole thing (on driver and on the yarn) properly?</p>