title: setJars() method in running Spark SQL app from IDE
body: <p>I am creating a Spark Sql Application and I want to run it on remote spark cluster from my local machine with my IDE. I know that I should set some option when I create SparkConf Object, smth like this:</p>
  
  <pre><code>SparkConf conf = new SparkConf()
  .setMaster("spark://SPARK-MASTER-ADDRESS:7077")
  .set("spark.driver.host","my local IP Address")
  .setJars(new String[]{"build\\libs\\spark-test-1.0-SNAPSHOT.jar"})
  .setAppName("APP-NAME");
  </code></pre>
  
  <p><strong>It's working from IDE and every thing is OK</strong>,</p>
  
  <p>but my questions are:</p>
  
  <p><strong>1)</strong> Do I need to rebuild the jar file of my app and set it's path to setJars method, every time I change anything? I saw that in some Forums had been said: you will need to build the jar every time you change anything. but It looks a little hard to rebuild app's jar file every time. Is there a better way for that?</p>
  
  <p><strong>2)</strong> Why is it sometimes not necessary to use setJars method, although I run the program through IDE ? For Example, When I do not use <strong>lambda function</strong> in my code there is no need to setjars function. Just  Assume I have a class of person that have two field: CustomerNo, AccountNo. When I use lamba function in my code like this (personDS is a dataset of person object):</p>
  
  <pre><code>personDS.filter(f -&gt; f.getCustomerNo().equals("001")).show();
  </code></pre>
  
  <p>the following error occurs:</p>
  
  <pre><code>java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
  </code></pre>
  
  <p>but when I don't use lamba function in my code like this:</p>
  
  <pre><code>personDS.filter(col("customerNo").equalTo(001)).show();
  </code></pre>
  
  <p>No Error Occurs. So, Why is this happend? Why I have to use setJars when I use lambda function? When I should use setJars and when not? </p>