title: Spark job fails when using checkpointing if a class change in the job
body: <p>I have my application jar sitting in HDFS which defines long-running Spark Streaming job and I am using checkpoint dir also in HDFS. Every time I have any changes to the job, I go delete that jar and upload a new one.</p>
  
  <p>Now if I upload a new jar and delete checkpoint directory it works fine. But if I don't delete the checkpoint directory, I get an error like:</p>
  
  <blockquote>
    <p>imestamp="2016-05-13T18:49:47,887+0000",level="WARN",threadName="main",logger="org.apache.spark.streaming.CheckpointReader",message="Error reading checkpoint from file hdfs://myCheckpoints/application-1/checkpoint-1463165355000",exception=<strong>"java.io.InvalidClassException: some.package.defined.here.ConcreteClass; local class incompatible: stream classdesc serialVersionUID = -7808345595732501156, local class serialVersionUID = 1574855058137843618</strong></p>
  </blockquote>
  
  <p>I have changed the 'ConcreteClass' from my last implementation and that's what's causing the issue.</p>
  
  <p>I have 2 main questions:</p>
  
  <ol>
  <li><strong>How to fix this?</strong> I know adding <code>private static long serialVersionUID = 1113799434508676095L;</code> might fix it, but I don't want to add this to all classes, since any class can change between current and next version. Anything better?</li>
  <li><strong>What all does checkpoint directory store?</strong> Does it store all classes from previous jar? Or just their name and serialVersionUID? <a href="https://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing" rel="nofollow">This</a> doesn't give much detail on internals of checkpointing. It's size is only ~ 6Mb. </li>
  </ol>
  
  <p>Appreciate any help.</p>